Service evaluation method and device and electronic equipment

Abstract

The embodiment of the invention discloses a service evaluation method and device and electronic equipment. The method comprises the following steps: determining a first analog terminal and a second analog terminal which are used for simulating a call, and obtaining a target service identifier corresponding to a target service requested to be started by the first analog terminal in a simulated call connection process by establishing a simulated call connection between the first analog terminal and the second analog terminal, an actual service result corresponding to the target service identifier is generated for a second simulation terminal in the simulation call connection process; and determining an expected service result corresponding to the target service identifier, and evaluating the target service based on the expected service result and the actual service result to obtain an evaluation result corresponding to the target service. Therefore, the target service is automatically evaluated by simulating the call, the evaluation accuracy can be improved, and the success rate of service implementation can be improved by using the evaluated target service during the real call, so that the accuracy of service implementation is effectively guaranteed.

Description

Technical Field

[0001] This invention relates to the field of communication technology, and in particular to a service evaluation method, apparatus, and electronic device. Background Technology

[0002] With the rapid development of 5G technology, many new features and services have been added to the field of 5G calling. These features and services can be used during calls, bringing users a more interesting, vivid, and diverse calling experience.

[0003] Currently, after the calling device initiates a functional service during a new 5G call, the called device can implement that service. However, the called device is unaware of the specific initiation details of the functional service, which may lead to a mismatch between the service implementation and the initiation, making it impossible to guarantee the accuracy of the functional service implementation. Therefore, a service evaluation scheme is needed to ensure the accuracy of the functional service implementation. Summary of the Invention

[0004] The purpose of this invention is to provide a technical solution that can evaluate functional services before they are deployed for application, thereby ensuring the accuracy of the functional service implementation.

[0005] To solve the above-mentioned technical problems, the embodiments of the present invention are implemented as follows: In a first aspect, an embodiment of the present invention provides a service evaluation method, the method comprising: A first and a second analog terminal are identified for simulating a call, and an analog call connection is established between the first and the second analog terminals. Obtain the target service identifier corresponding to the target service requested to be opened by the first simulated terminal during the simulated call connection, and determine the expected service result corresponding to the target service identifier; Obtain the actual service result corresponding to the target service identifier generated for the second simulated terminal during the simulated call connection process; Based on the expected service results and the actual service results, the target service is evaluated to obtain the evaluation results corresponding to the target service.

[0006] Secondly, embodiments of the present invention provide a service evaluation device, the device comprising: The call module determines a first and a second simulated terminal for simulating a call, and establishes a simulated call connection between the first and the second simulated terminals. The processing module obtains the target service identifier corresponding to the target service requested to be opened by the first simulated terminal during the simulated call connection process, and determines the expected service result associated with the target service based on the target service identifier; and obtains the actual service result of the target service on the second simulated terminal during the simulated call connection process. The evaluation module evaluates the target service based on the expected service result and the actual service result, and obtains the evaluation result corresponding to the target service.

[0007] Thirdly, embodiments of the present invention provide an electronic device, including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the service evaluation method provided in the above embodiments.

[0008] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of the service evaluation method provided in the above embodiments.

[0009] Fifthly, embodiments of the present invention provide a computer program product, including a computer program that, when executed by a processor, implements the steps of the service evaluation method provided in the above embodiments. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a flowchart illustrating a service evaluation method according to an embodiment of the present invention. Figure 2 This is a schematic diagram of the structure of an analog terminal provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of a process for evaluating a target service, provided as another embodiment of the present invention. Figure 4 A schematic diagram of a service evaluation scenario provided in another embodiment of the present invention; Figure 5 A logical diagram illustrating a service evaluation process provided in another embodiment of the present invention; Figure 6 A network architecture diagram of a call process is provided in another embodiment of the present invention; Figure 7 A schematic diagram of the structure of a service evaluation device provided in another embodiment of the present invention; Figure 8 This is a schematic diagram of the structure of an electronic device provided in another embodiment of the present invention. Detailed Implementation

[0012] This invention provides a service evaluation method, apparatus, and electronic device.

[0013] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this invention.

[0014] This specification provides a service evaluation method, apparatus, and electronic device. With the rapid development of 5G new calling technology, many new functional services have been added to the 5G new calling field, such as intelligent translation, background replacement, subtitle synthesis, and video synthesis. When the calling device and the called device use these functional services during a 5G new call, they can achieve real-time immersive interactive calls, bringing a more interesting, vivid, and diverse calling experience to device users. However, currently, after the calling device initiates a functional service during a 5G new call, the called device is unaware of the specific initiation of the functional service, which may lead to a mismatch between service implementation and service initiation, making it impossible to guarantee the accuracy of the functional service implementation.

[0015] To address this, this specification provides a better service evaluation scheme, establishing a first and second simulated terminal for simulating calls. By establishing a simulated call connection between the first and second simulated terminals, the target service identifier corresponding to the target service requested by the first simulated terminal during the simulated call connection is obtained, and the expected service result corresponding to the target service identifier is determined. The actual service result corresponding to the target service identifier generated for the second simulated terminal during the simulated call connection is obtained. Then, based on the expected and actual service results, the target service is evaluated to obtain the evaluation result corresponding to the target service. In this way, deploying the evaluated target service into the application can improve the success rate of service implementation, thereby effectively ensuring the accuracy of functional service implementation.

[0016] Figure 1 This is a flowchart illustrating a service evaluation method according to an embodiment of the present invention. Figure 1As shown, the execution subject of this method can be a terminal device or a server. The terminal device can be a mobile terminal device such as a mobile phone, tablet, or smartwatch, or a terminal device such as a computer. The server can be a standalone server or a server cluster composed of multiple servers. Specifically, the method may include the following steps: In step S102, a first analog terminal and a second analog terminal for simulating a call are determined, and an analog call connection is established between the first analog terminal and the second analog terminal.

[0017] The simulated terminal is a device that simulates the functions and interactive characteristics of a real calling device, providing controllable and repeatable verification conditions for service evaluation. A real calling device is a mobile terminal device with calling functionality and capable of establishing a normal call connection, such as a mobile phone, tablet, or smartwatch. The first and second simulated terminals are two identical simulated terminals, serving as the calling terminal and the called terminal respectively during the call. The calling terminal is the device or user terminal that actively initiates the communication request, typically referring to the party making the call; the called terminal is the device or user terminal that receives and responds to the communication request, typically referring to the party answering the call.

[0018] In implementation, the first analog terminal can be used as the calling terminal, and the second analog terminal as the called terminal. Call conditions can be preset on the first analog terminal, triggering a call to the second analog terminal. Alternatively, the evaluation personnel can manually initiate a call on the first analog terminal to establish a simulated call connection between the two terminals. This simulated call connection refers to simulating the entire process of a real call, from dialing to hanging up, in a virtual call environment. The simulated call connection can be a voice call connection or a video call connection, replicating the connection logic, signal interaction, and user experience of a real call. Of course, the second analog terminal can also be used as the calling terminal, and the first analog terminal as the called terminal. Subsequently, a target service can be initiated on the second analog terminal, generating an actual service result for the first analog terminal. This embodiment does not limit this approach.

[0019] Optionally, a simulated call connection between the first and second simulated terminals can be established via a 183 / update / 200 signaling interaction process. The specific process is as follows: The first simulated terminal indicates the media formats it supports (such as codec type, sampling rate, etc.) to the second simulated terminal via signaling message 183; based on the formats supported by the second simulated terminal, the first simulated terminal proposes a specific media format scheme (such as a specific codec or transmission parameters) via message Update. This message contains negotiation details for the second simulated terminal's reference. After receiving the proposal from the first simulated terminal, the second simulated terminal indicates its agreement via message 200. At this point, both parties have reached an agreement and can continue with subsequent resource reservation and simulated call connection establishment.

[0020] Figure 2 This is a schematic diagram of the structure of an analog terminal provided in one embodiment of this specification. For example... Figure 2 As shown, this simulated terminal can be either a first simulated terminal or a second simulated terminal. The first simulated terminal includes an analog call pager, a signal transmitter and receiver, a message protocol converter, and a wireless base station simulator. Similarly, the second simulated terminal can also include an analog call pager, a signal transmitter and receiver, a message protocol converter, and a wireless base station simulator. The analog call pager is used to make or receive calls; the signal transmitter and receiver is used to convert between voice and digital signals; the message protocol converter is used to convert digital signals from the terminal side into core network protocol information to enable core network message interaction; and the wireless base station simulator is used to simulate the functions of a wireless network base station, enabling the processing, forwarding, and transmission of voice, image, and video data.

[0021] In step S104, the target service identifier corresponding to the target service requested to be opened by the first simulated terminal during the simulated call connection is obtained, and the expected service result corresponding to the target service identifier is determined.

[0022] In this context, target services can be understood as value-added services that overlay new channel data or technological capabilities on top of traditional audio and video calls, enabling multimedia information interaction, extended sensory experiences, and innovative business scenarios. Examples of target services include screen sharing, content sharing, interactive fun, background replacement, virtual avatars, intelligent customer service, or intelligent translation.

[0023] In addition, the target service identifier can be used to characterize the type and category of the target service, enabling precise identification of user intent. The service identifier can include two parts: one part describes the service type, such as interactive fun service or background replacement service; the other part describes the service category within that specific service type, such as background 1 in a background replacement service or background 2 in a background replacement service. The expected service result corresponding to the target service identifier refers to the ideal effect that the target service pointed to by the target service identifier is expected to achieve.

[0024] In implementation, the first simulated terminal can actively activate the target service through function buttons or menus in the call interface; it can also directly activate the target service through voice commands; or it can be pre-configured with trigger conditions to automatically activate the target service during a call. After detecting that the first simulated terminal requests to activate the target service, the target service identifier corresponding to the target service can be obtained, and the expected service result corresponding to the target service identifier can be determined.

[0025] In step S106, the actual service result corresponding to the target service identifier generated for the second simulated terminal during the simulated call connection process is obtained.

[0026] In implementation, a simulated call connection is established between the first and second simulated terminals. The target service initiated by the first simulated terminal requires interaction with the second simulated terminal to be completed. For example, the intelligent translation service needs to first recognize the speech on the second simulated terminal before it can be translated. Therefore, the actual service result generated for the second simulated terminal refers to the service result determined for the target service based on the interaction between the first and second simulated terminals.

[0027] In step S108, the target service is evaluated based on the expected service results and the actual service results to obtain the evaluation results corresponding to the target service.

[0028] In implementation, the expected service result is the ideal result that the target service can achieve. The expected service result can be used as an evaluation standard. By judging whether the actual service result meets this evaluation standard, the evaluation result corresponding to the target service can be obtained.

[0029] For example, the evaluation criteria can be broken down into multiple evaluation dimensions with corresponding scoring standards. Based on the correspondence between the evaluation dimensions and the scoring standards, the dimensional scores of the actual service results under multiple evaluation dimensions are calculated. The final evaluation score of the actual service results is determined based on the multiple dimensional scores, and the evaluation result of the target service is determined based on the final evaluation score. Of course, the actual service results can also be directly compared with the expected service results, and the evaluation result of the target service can be determined based on the comparison results. This embodiment does not limit this approach.

[0030] This specification provides a service evaluation method. It identifies a first and second simulated terminal for simulating a call. By establishing a simulated call connection between the first and second simulated terminals, it obtains the target service identifier corresponding to the target service requested by the first simulated terminal during the simulated call connection, and the actual service result corresponding to the target service identifier generated for the second simulated terminal during the simulated call connection. It then determines the expected service result corresponding to the target service identifier and evaluates the target service based on the expected and actual service results to obtain the evaluation result corresponding to the target service. Accordingly, simulating a call between the calling and called parties, and automatically evaluating the functional services used in the call, avoids the subjective influence of human evaluation and improves the accuracy of the evaluation. Furthermore, using the evaluated functional services during a real call can increase the success rate of service implementation, thereby effectively ensuring the accuracy of service implementation.

[0031] In practical applications, there may be multiple ways to process step S104, which involves obtaining the target service identifier corresponding to the target service requested by the first simulated terminal during the simulated call connection. In one optional processing method, the processing method may specifically include the following steps: Step A2: During the simulated call connection, the first simulated terminal sends a target service request to the control network element to request the activation of the target service, so that the control network element sends the target service request to the media network element. Step A4: The receiving media network element extracts and sends the service identifier from the target service request, and uses the received service identifier as the target service identifier corresponding to the target service requested to be opened by the first simulated terminal during the simulated call connection process.

[0032] The control network element, as a core component of the communication network, is responsible for session management, policy control, signaling processing, and routing decisions, ensuring the efficient and reliable establishment, maintenance, and release of calls. The media network element, as a core component in the communication network for processing media streams, is responsible for the transmission, processing, and quality assurance of media streams. In this embodiment, interaction between the first analog terminal and the second analog terminal can be achieved based on the control network element and the media network element.

[0033] In implementation, the first simulated terminal can initiate a call request to the second simulated terminal to establish a simulated call connection. When the first simulated terminal needs to start a target service, it sends a target service request to the control network element; the control network element forwards the target service request to the media network element to request the media network element to execute the target service, wherein the target service request carries a target service identifier; the media network element can extract the target service identifier from the target service request and obtain the media content required to execute the target service based on the target service identifier; and obtain the target service identifier corresponding to the target service from the media network element.

[0034] It should be noted that in a communication network, the Application Server (AS) is the core component responsible for starting and executing various value-added services under the control of business logic, and can be used to support the implementation and expansion of value-added services. Optionally, multiple application servers can be deployed for the first simulated terminal, with different application servers managing different functional services. After detecting that the first simulated terminal has initiated a target service, the application server responsible for the target service can send a target service request to the control network element; the control network element returns a request response to the application server to inform it that the request has been received, and can then send the target service request to the media network element.

[0035] In addition, when a media network element obtains the media content required to execute a target service, it can first check whether the media content required to execute the target service is stored in the media library associated with the media network element. If not, it downloads the media content required to execute the target service from the media capability platform in the communication network according to the download address and target service identifier contained in the target service request sent by the control network element. The downloaded media content is associated with the target service identifier. Furthermore, the downloaded media content is saved to the media library associated with the media network element.

[0036] This specification provides a service evaluation method that uses control network elements, media network elements, and simulated terminals to conduct call interactions, enabling the simulated call process to accurately reproduce the real call process. Service evaluation is performed during the simulated call, and the evaluated functions are more adapted to the real call process, improving the success rate of service functions in real calls and thus effectively ensuring the accuracy of service implementation.

[0037] In practical applications, there may be multiple ways to determine the expected service result corresponding to the target service identifier in step S104. One optional processing method may specifically include the following steps: Step B2: Based on the target service identifier, query the service result corresponding to the target service identifier from the preset service result database, and use the queried service result as the expected service result.

[0038] The preset service result database contains a mapping relationship between service identifiers and service results. The service results in the database represent the expected service results that various functional services are designed to achieve. Therefore, after the first simulated terminal initiates a target service, the service results mapped to the target service identifier retrieved from the preset service result database can be used as the expected service result corresponding to the target service identifier.

[0039] Step B4: If no service result corresponding to the target service identifier is found, search for at least one service result related to the target service in the media library associated with the media network element, synchronize at least one service result to the preset service result library, and re-execute the query operation based on the target service identifier.

[0040] In practice, if the target service is used for the first time, the preset service result library may not contain service results related to the target service. Therefore, it is not possible to query the service results mapped to the target service identifier from the preset result library.

[0041] In this case, based on at least one service identifier involved in the target service, the service result corresponding to at least one service identifier can be queried in the media library associated with the media network element, and the mapping relationship between at least one service identifier and the service result can be synchronized to the preset service result library to update the preset service result library. Afterwards, the query operation in step B2 can be re-executed.

[0042] It should be noted that when the service result mapped to the target service identifier cannot be found in the preset service result database, the mapping relationship between the service identifier and the service result can also be synchronized from other locations, such as a database containing all information of the target service, or manual evaluation records containing the mapping relationship between the service identifier and the service result.

[0043] In practical applications, step S108, which evaluates the target service based on the expected service result and the actual service result, can be implemented in several ways. One possible implementation is as follows: Figure 3 As shown, the specific steps may include: In step S302, the actual service result is compared with the expected service result to determine whether the expected service result and the actual service result are the same. In step S304, if the actual service result is the same as the expected service result, then the evaluation result corresponding to the target service is recorded as passed. In step S306, if the actual service result is different from the expected service result, the failure to pass the evaluation is recorded in the evaluation result corresponding to the target service.

[0044] As mentioned earlier, the expected service result is an evaluation standard that compares the actual service result with the expected service result. By judging whether the expected service result and the actual service result are the same, it can be determined whether the actual service result meets the evaluation standard.

[0045] If the actual service result is the same as the expected service result, it indicates that the actual service result meets the evaluation criteria, and the target service can be determined to have passed the evaluation. The record of passing the evaluation will be entered in the corresponding evaluation results for the target service. If the actual service result differs from the expected service result, it indicates that the actual service result does not meet the evaluation criteria, and the target service can be determined to have failed the evaluation. The record of failing the evaluation will be entered in the corresponding evaluation results for the target service.

[0046] Of course, other methods can also be used to evaluate the target service. For example, compare the statistical distribution (such as mean, variance, median, quantile) of the actual service results with that of the expected service results, and evaluate the target service based on the statistical distribution to obtain the evaluation results corresponding to the target service; check the correlation between the actual service results and the expected service results (such as the Pearson correlation coefficient), and verify whether the two change synchronously to detect the target service, etc.

[0047] Based on this, further processing can be performed on the first and second simulated terminals according to the evaluation results. Depending on the different evaluation results, the specific processing methods adopted for the first and second simulated terminals may include the following two situations. In one scenario, if the evaluation result records that the evaluation was passed, the simulated call between the first and second simulated terminals continues. In this case, a pass signal can be sent back to the second simulated terminal to continue the simulated call process.

[0048] In another scenario, when the evaluation result records a failure, the system retrieves the preset failure handling rules for the target service and executes the specified operation within those rules. The specified operation includes continuing the simulated call or ending the simulated call. In this case, a failure signal can be sent to the second simulated terminal to trigger it to retrieve the preset failure handling rules. The failure handling rules may include multiple optional operations, such as continuing the simulated call or ending the simulated call. Each optional operation has a corresponding selection state, which can be selected or unselected. The optional operation with the selected state is used as the specified operation in the failure handling rules.

[0049] This specification provides a service evaluation method that receives the actual service results corresponding to a target service identifier, determines the expected service results corresponding to the target service identifier based on a preset result database, and automatically evaluates the target service based on the expected and actual service results. This avoids the influence of subjective judgment by humans and improves the accuracy of service evaluation. Furthermore, it automatically controls the subsequent processing steps of the simulated call process based on the evaluation results, making the entire evaluation process free from human intervention, thus saving a significant amount of manpower and improving the efficiency of service evaluation.

[0050] Figure 4 A scenario logic diagram of a service evaluation method provided in another embodiment of the present invention is shown below. Figure 4 As shown, in the application scenario, the target service is the background replacement service. The evaluation process for the background replacement service can specifically include the following steps: In step S402, the first analog terminal initiates a call request to the second analog terminal to establish an analog call connection between the first analog terminal and the second analog terminal; In step S404, during the simulated call, the application server corresponding to the background replacement service completes the pre-triggering of the background replacement service according to the preset triggering logic; In step S406, the application server corresponding to the background replacement service sends a background replacement request to the control network element when it determines that the first simulated terminal has initiated the background replacement service. In step S408, the control network element returns a request response to the application server corresponding to the background replacement service, indicating that the background replacement request has been received; In step S410, the control network element sends a background replacement request to the media network element. The background replacement request includes the media ID and the content download address. In step S412, if the media network element determines that the media content required for background replacement does not exist in its associated media library, it downloads the media content associated with the media ID required for background replacement from the media capability platform according to the content download address in the background replacement request, and saves it to the media library. In step S414, the media network element sends the media ID to the simulation test module; In step S416, the media network element performs background replacement processing based on the media content required for background replacement, obtains the actual video stream corresponding to the media ID, and sends the obtained actual video stream to the second analog terminal; In step S418, the second simulation terminal sends the received actual video stream to the simulation test module; In step S420, the simulation test module determines the expected video stream corresponding to the media ID, and evaluates the background replacement service based on the expected video stream and the actual video stream to obtain the evaluation result of the background replacement service.

[0051] It should be noted that, in this embodiment, the first simulated terminal, the second simulated terminal, and the simulation testing module can be integrated into a simulation testing system. This simulation testing system achieves automatic evaluation of the target service by interacting with control network elements, media network elements, etc. Furthermore, the simulation testing system can also include multiple simulated terminals, supporting the simultaneous establishment of multiple simulated call connections to enable simultaneous evaluation of multiple functional services, thereby improving the efficiency of service evaluation.

[0052] Figure 5 A logical diagram of a service evaluation process provided in another embodiment of the present invention, as shown below. Figure 5 As shown, the execution entity of this method can be a simulation test module, and step S420 can specifically include the following steps S502 to S512: In step S502, the media ID sent by the media network element is received; In step S504, a query is performed from the preset service result database to determine if the desired video stream corresponding to the media ID exists; If it does not exist, proceed to step S506, synchronize the video streams related to the background replacement service in the media library associated with the media network element to the preset service result library, and re-execute step S504. If it exists, proceed to step S508 to determine whether the actual video stream is consistent with the expected video stream; If they match, proceed to step S510 to confirm that the background replacement service evaluation has passed, and send a pass signal to the second analog terminal so that the first and second analog terminals can continue to perform the analog call. If there is a discrepancy, proceed to step S512 to determine that the background replacement service evaluation has failed. Send a failure signal to the second analog terminal. In response to the failure signal, obtain the preset failure handling rules for the background replacement service and execute the specified operation in the failure handling rules. The specified operation can be to continue the analog call or end the analog call.

[0053] It should be noted that in this embodiment, multiple service evaluations can be performed on a single target service. For example, the target service can be initiated periodically, or the evaluation personnel can manually initiate the target service multiple times. When the pass rate of multiple service evaluations is higher than the deployment threshold, the target service can be deployed to the application, making it usable during real calls. During a real call, users can initiate the service through the dial pad in the video / voice call UI, view the service information directly through the terminal call interface, and interact with the service via touchscreen.

[0054] Figure 6 A network architecture diagram of a call process is provided in another embodiment of the present invention, with reference to... Figure 6 5G New Voice resides at the application and service layers of the operator's core network and consists of 5G New Voice equipment, 5G New Voice service AS, media capability platform, and operation management platform. The New Voice network primarily comprises New Voice capability network elements and a New Voice media plane. The 5G New Voice capability network elements are connected to the VoLTE AS and S-CSCF, providing 5G New Voice audio, video, and IMS DC control functions. The New Voice media plane is connected to the VoLTE core network media plane and can perform various functional services provided by 5G New Voice.

[0055] In practical applications, the simulation testing system collaborates with the 5G core network and multimedia subsystem to establish, transmit, and control calls. The simulation testing system includes a simulated terminal and a simulation testing module. The simulated terminal simulates a user's use of the functional services in a new 5G call, responsible for signal transmission and reception and user interaction. The simulation testing module is responsible for evaluating the functional services initiated by the user. The 5G core network is responsible for network connectivity and resource management. The 5G core network includes the User Plane Function (UPF) and the Access and Mobility Management Function (AMF). The UPF and AMF respectively handle user plane data processing and mobility management tasks. The multimedia subsystem is responsible for multimedia session control and service provision. The multimedia subsystem includes a new call platform, a core network element (PacketSwitched Core Function, PSBC), and a call session control function (CSCF). The new call platform is responsible for the interactive management of functional services during 5G new calls. The PSBC is responsible for handling user access requests, supporting multiple access technologies (such as GPRS / WiFi), and managing users and traditional networks. The CSCF is mainly responsible for handling signaling control during multimedia call sessions.

[0056] This specification provides a service evaluation method, which includes a simulated terminal that simulates a main background terminal making a call. A simulation test module is designed to receive media IDs sent by media network elements and query the expected video stream corresponding to the media ID from a preset service result database. When the actual video stream is received, a comparison between the actual and expected video streams is automatically performed to achieve automatic evaluation of the background replacement service. This adds a media video stream storage and comparison function while ensuring the normal and complete operation of the business process. It solves the problems of the inability to automatically evaluate 5G new call services and quickly perceive the correctness of 5G new call equipment functionality, enabling rapid detection of equipment malfunctions and optimizing 5G new call equipment maintenance methods, thereby improving the success rate of 5G new call services.

[0057] The above describes the service evaluation method provided in the embodiments of this specification. Based on the same idea, the embodiments of this specification also provide a service evaluation device, such as... Figure 7 As shown.

[0058] The service evaluation device includes: a call module 701, a processing module 702, and an evaluation module 703, wherein: The call module 701 determines a first simulated terminal and a second simulated terminal for simulating a call, and establishes a simulated call connection between the first simulated terminal and the second simulated terminal. The processing module 702 obtains the target service identifier corresponding to the target service requested to be opened by the first simulated terminal during the simulated call connection process, and determines the expected service result associated with the target service based on the target service identifier; and obtains the actual service result of the target service on the second simulated terminal during the simulated call connection process. The evaluation module 703 evaluates the target service based on the expected service result and the actual service result, and obtains the evaluation result corresponding to the target service.

[0059] In this embodiment of the specification, the processing module 702 is further configured to: During the simulated call connection, the first simulated terminal sends a target service request to the control network element to request the activation of the target service, so that the control network element sends the target service request to the media network element. The service identifier extracted and sent by the media network element from the target service request is received, and the received service identifier is used as the target service identifier corresponding to the target service requested to be opened by the first simulated terminal during the simulated call connection process.

[0060] In this embodiment of the specification, the processing module 702 is further configured to: Based on the target service identifier, query the service result corresponding to the target service identifier from the preset service result database, and use the queried service result as the expected service result; If no service result corresponding to the target service identifier is found, at least one service result related to the target service is searched from the media library corresponding to the media network element, the at least one service result is synchronized to the preset service result library, and the query operation is re-executed based on the target service identifier.

[0061] In this embodiment of the specification, the evaluation module 703 includes a judgment unit, which is used for: The actual service result is compared with the expected service result to determine whether the expected service result and the actual service result are the same; If the actual service result is the same as the expected service result, then the evaluation result corresponding to the target service is recorded as "passed". If the actual service result differs from the expected service result, then the evaluation result corresponding to the target service is recorded as "failed evaluation".

[0062] In this embodiment of the specification, the processing module 703 includes a feedback unit, which is used for: When the evaluation result records that the evaluation is passed, the simulated call between the first simulated terminal and the second simulated terminal continues. When the evaluation result records a failure, the preset failure handling rules for the target service are obtained, and the specified operation in the failure handling rules is executed. The specified operation includes continuing to perform the simulated call or ending the simulated call.

[0063] In the embodiments described in this specification, the first simulated terminal includes a simulated call pager, a signal transmitter and receiver, a message protocol converter, and a wireless base station simulator; the second simulated terminal includes a simulated call pager, a signal transmitter and receiver, a message protocol converter, and a wireless base station simulator.

[0064] This specification provides a service evaluation device that determines a first simulated terminal and a second simulated terminal for simulating a call. By establishing a simulated call connection between the first and second simulated terminals, it obtains the target service identifier corresponding to the target service requested by the first simulated terminal during the simulated call connection, and the actual service result corresponding to the target service identifier generated for the second simulated terminal during the simulated call connection. It then determines the expected service result corresponding to the target service identifier and evaluates the target service based on the expected and actual service results to obtain the evaluation result corresponding to the target service. Accordingly, by simulating a call between the calling and called parties and automatically evaluating the functional services used in the call, it avoids the subjective influence of human evaluation and improves the accuracy of the evaluation. Furthermore, using the evaluated functional services during a real call can increase the success rate of service implementation, thereby effectively ensuring the accuracy of service implementation.

[0065] The above describes the service evaluation device provided in the embodiments of this specification. Based on the same idea, the embodiments of this specification also provide an electronic device, such as... Figure 8 As shown.

[0066] The electronic device can provide a terminal device or server, etc., for the above embodiments.

[0067] Electronic devices can vary considerably due to differences in configuration or performance. They may include one or more processors 801 and memories 802, with the memory 802 storing one or more application programs or data. The memory 802 may be temporary or persistent storage. The application programs stored in the memory 802 may include one or more modules (not shown), each module including a series of computer-executable instructions for the electronic device. Furthermore, the processor 801 may be configured to communicate with the memory 802 and execute the series of computer-executable instructions stored in the memory 802 on the electronic device. The electronic device may also include one or more power supplies 803, one or more wired or wireless network interfaces 804, one or more input / output interfaces 805, and one or more keyboards 806.

[0068] Specifically, in this embodiment, the electronic device includes a memory and one or more programs, wherein one or more programs are stored in the memory, and one or more programs may include one or more modules, and each module may include a series of computer-executable instructions for use in the electronic device, and is configured to be executed by one or more processors. The one or more programs include computer-executable instructions for performing the following: A first and a second analog terminal are identified for simulating a call, and an analog call connection is established between the first and the second analog terminals. Obtain the target service identifier corresponding to the target service requested to be opened by the first simulated terminal during the simulated call connection, and determine the expected service result corresponding to the target service identifier; Obtain the actual service result corresponding to the target service identifier generated for the second simulated terminal during the simulated call connection process; Based on the expected service results and the actual service results, the target service is evaluated to obtain the evaluation results corresponding to the target service.

[0069] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the electronic device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0070] This specification provides an electronic device that determines a first simulated terminal and a second simulated terminal for simulating a call. By establishing a simulated call connection between the first and second simulated terminals, it obtains the target service identifier corresponding to the target service requested by the first simulated terminal during the simulated call connection, and the actual service result corresponding to the target service identifier generated for the second simulated terminal during the simulated call connection. It then determines the expected service result corresponding to the target service identifier and evaluates the target service based on the expected and actual service results to obtain the evaluation result corresponding to the target service. Accordingly, simulating a call between the calling and called parties, and automatically evaluating the functional services used in the call, avoids the subjective influence of human evaluation and improves the accuracy of the evaluation. Furthermore, using the evaluated functional services during a real call can increase the success rate of service implementation, thereby effectively ensuring the accuracy of service implementation.

[0071] Furthermore, based on the above Figures 1 to 6 The method shown in this specification, along with one or more embodiments, also provides a storage medium for storing computer-executable instruction information. In one specific embodiment, the storage medium can be a USB flash drive, optical disc, hard disk, etc. When the computer-executable instruction information stored in the storage medium is executed by a processor, it can achieve the following process: A first and a second analog terminal are identified for simulating a call, and an analog call connection is established between the first and the second analog terminals. Obtain the target service identifier corresponding to the target service requested to be opened by the first simulated terminal during the simulated call connection, and determine the expected service result corresponding to the target service identifier; Obtain the actual service result corresponding to the target service identifier generated for the second simulated terminal during the simulated call connection process; Based on the expected service results and the actual service results, the target service is evaluated to obtain the evaluation results corresponding to the target service.

[0072] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the above-described storage medium embodiment is basically similar to the method embodiment, so the description is relatively simple; relevant parts can be referred to the description of the method embodiment.

[0073] This specification provides a storage medium that determines a first and a second simulated terminal for simulating a call. By establishing a simulated call connection between the first and second simulated terminals, it obtains the target service identifier corresponding to the target service requested by the first simulated terminal during the simulated call connection, and the actual service result corresponding to the target service identifier generated for the second simulated terminal during the simulated call connection. It then determines the expected service result corresponding to the target service identifier and evaluates the target service based on the expected and actual service results to obtain the evaluation result corresponding to the target service. Accordingly, simulating a call between the calling and called parties, and automatically evaluating the functional services used in the call, avoids the subjective influence of human evaluation and improves the accuracy of the evaluation. Furthermore, using the evaluated functional services during a real call can improve the success rate of service implementation, thereby effectively ensuring the accuracy of service implementation.

[0074] Furthermore, based on the above Figures 1 to 6 The method shown in this specification, along with one or more embodiments, also provides a computer program product including a computer program that, when executed by a processor, performs the following process: A first and a second analog terminal are identified for simulating a call, and an analog call connection is established between the first and the second analog terminals. Obtain the target service identifier corresponding to the target service requested to be opened by the first simulated terminal during the simulated call connection, and determine the expected service result corresponding to the target service identifier; Obtain the actual service result corresponding to the target service identifier generated for the second simulated terminal during the simulated call connection process; Based on the expected service results and the actual service results, the target service is evaluated to obtain the evaluation results corresponding to the target service.

[0075] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the above-described embodiment of a computer program product is relatively simple in description because it is fundamentally similar to the method embodiment; relevant parts can be referred to the description of the method embodiment.

[0076] This specification provides a computer program product that determines a first and a second simulated terminal for simulating a call. By establishing a simulated call connection between the first and second simulated terminals, it obtains the target service identifier corresponding to the target service requested by the first simulated terminal during the simulated call connection, and the actual service result corresponding to the target service identifier generated for the second simulated terminal during the simulated call connection. It then determines the expected service result corresponding to the target service identifier and evaluates the target service based on the expected and actual service results to obtain the evaluation result corresponding to the target service. Accordingly, simulating a call between the calling and called parties, and automatically evaluating the functional services used in the call, avoids the subjective influence of human evaluation and improves the accuracy of the evaluation. Furthermore, using the evaluated functional services during a real call can increase the success rate of service implementation, thereby effectively ensuring the accuracy of service implementation.

[0077] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0078] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must also be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0079] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0080] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0081] For ease of description, the above apparatus is described by dividing it into various functional units. Of course, when implementing one or more embodiments of this specification, the functions of each unit can be implemented in one or more software and / or hardware.

[0082] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, one or more embodiments of this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0083] Embodiments in this specification are described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable parallel device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable parallel device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0084] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable fraud device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0085] These computer program instructions can also be loaded onto a computer or other programmable device, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0086] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0087] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0088] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0089] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0090] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, one or more embodiments of this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0091] One or more embodiments of this specification can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a particular task or implement a particular abstract data type. One or more embodiments of this specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0092] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0093] The above description is merely an embodiment of this specification and is not intended to limit this document. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this specification.

Claims

1. A service evaluation method characterized by comprising: The method comprises: determining a first simulation terminal and a second simulation terminal for simulating a call, and establishing a simulation call connection between the first simulation terminal and the second simulation terminal; obtaining a target service identifier corresponding to a target service requested to be started by the first simulation terminal during the simulation call connection process, and determining an expected service result corresponding to the target service identifier; obtaining an actual service result corresponding to the target service identifier generated for the second simulation terminal during the simulation call connection process; based on the expected service result and the actual service result, evaluating the target service to obtain an evaluation result corresponding to the target service.

2. The method of claim 1, wherein, The method comprises: during the simulation call connection process, sending a target service request for requesting to start the target service to a control network element through the first simulation terminal, so that the control network element sends the target service request to a media network element; receiving a service identifier extracted and sent by the media network element from the target service request, and taking the received service identifier as the target service identifier corresponding to the target service requested to be started by the first simulation terminal during the simulation call connection process.

3. The method of claim 1, wherein, The method comprises: based on the target service identifier, querying a service result corresponding to the target service identifier from a preset service result library, and taking the queried service result as the expected service result; if the service result corresponding to the target service identifier is not queried, finding at least one service result involved in the target service from a media library corresponding to the media network element, synchronizing the at least one service result to the preset service result library, and re-executing the query operation based on the target service identifier.

4. The method of claim 1, wherein, The method comprises: comparing the actual service result with the expected service result to determine whether the expected service result and the actual service result are the same; if the actual service result and the expected service result are the same, recording pass in the evaluation result corresponding to the target service; if the actual service result and the expected service result are different, recording fail in the evaluation result corresponding to the target service.

5. The method of claim 4, wherein, The method further comprises: when the evaluation result records pass, continuing to execute the simulation call between the first simulation terminal and the second simulation terminal; when the evaluation result records fail, obtaining a preset fail handling rule for the target service, and executing a specified operation in the fail handling rule, the specified operation comprising continuing to execute the simulation call or ending the simulation call.

6. The method according to any one of claims 1 to 5, characterized in that, The first simulation terminal comprises a simulation call annunciator, a signal sending receiver, a message protocol converter and a wireless base station simulator; and the second simulation terminal comprises a simulation call annunciator, a signal sending receiver, a message protocol converter and a wireless base station simulator.

7. A service evaluation device, characterized by comprising: The device comprises: a call module, which determines a first simulation terminal and a second simulation terminal for simulation call, and establishes a simulation call connection between the first simulation terminal and the second simulation terminal; a processing module, which acquires a target service identifier corresponding to a target service requested to be started by the first simulation terminal in the simulation call connection process, and determines an expected service result associated with the target service based on the target service identifier; and acquires an actual service result of the target service on the second simulation terminal in the simulation call connection process; an evaluation module, which evaluates the target service based on the expected service result and the actual service result, and obtains an evaluation result corresponding to the target service.

8. An electronic device, comprising: The computer program is stored on the computer readable storage medium and is executable on the processor, and when executed by the processor, the computer program implements the steps of the service evaluation method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer program is stored on the computer readable storage medium and is executable on the processor, and when executed by the processor, the computer program implements the steps of the service evaluation method according to any one of claims 1 to 6.

10. A computer program product, characterised in that, The computer program is stored on the computer readable storage medium and is executable on the processor, and when executed by the processor, the computer program implements the steps of the service evaluation method according to any one of claims 1 to 6.

Images