Method and terminal for automated evaluation of a virtual background

Abstract

The application provides a kind of virtual background automatic evaluation method and terminal, including background management module obtains original data and its background virtualization after being evaluated data, obtains evaluation data pair, and is configured with multiple frame rates for evaluation data pair, and evaluation task is issued to terminal management module;Terminal management module obtains the evaluation data pair required from background management module according to evaluation task and equipment list for evaluation;Terminal management module starts corresponding terminal equipment based on image similarity algorithm to evaluate data pair according to equipment list for virtual background effect evaluation.The application can adapt to different terminal equipment for background seeking effect evaluation, and the whole evaluation process only needs test personnel to realize through background management module group control and issue evaluation task to corresponding terminal equipment for automatic virtual background effect evaluation, which is convenient and fast, and effectively improves the virtual background effect evaluation efficiency of application.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of automation testing, in particular to a virtual background automatic evaluation method and a terminal. BACKGROUND

[0002] With the rapid development of the Internet and technological progress, in recent years, Internet live broadcast, online conference and other software have developed rapidly. Among them, many applications have carried out virtual background replacement functions, not only in live broadcast and online conference, but also in games. Virtual background replacement in software undoubtedly improves user experience, for example:

[0003] In the online conference scenario, virtual background replacement can be used to reduce the privacy exposure of the environment in which the individual is located;

[0004] In the entertainment live broadcast scenario, various funny scenes can be combined with the individual's image to achieve the effect of entertainment;

[0005] In the game scenario, the individual's image can interact with the game characters.

[0006] At present, the application scenarios of virtual background are numerous and belong to a continuous development process. Each manufacturer will introduce new frameworks and algorithms to optimize the effect of virtual background. Evaluating the effect of improvement has become the core work of R&D and test personnel.

[0007] The current evaluation test method mainly relies on manual evaluation. The real image of the person is captured through the camera, and the new picture is presented on the application after virtual background replacement. The effect of the generated picture is evaluated by manual evaluation, so as to provide the direction of effect adjustment for the R&D personnel. At the same time, the problem of computing power is involved. Such functions have high requirements for the performance of the device, and the R&D and test personnel also need to pay attention to and verify the performance of the evaluation process. The existing test evaluation method mainly has the following shortcomings:

[0008] Shortcoming one, virtual background business testing often needs to invest a large amount of manpower cost to evaluate the video picture. The most common test method is that the R&D and test personnel first list the test scenarios (such as: number of people, clothing color, background environment, action amplitude, etc.), usually a test list can reach hundreds, and a virtual background product often needs multiple versions of internal testing. If the R&D and test personnel manually simulate the content on the test list through the camera every time, it undoubtedly needs to spend a lot of time.

[0009] Second, in the above scenario, the virtual background business test, even if the tester has enough time to invest, in the iterative test process, the test results between two rounds are likely to be biased, even the same test package, the test results may be biased; The reason is that the test is performed according to the test list, but the image input picture may be different from the previous one, even if the same person wears the same clothes in the same position for testing, it is possible that the light is different between day and night, resulting in different virtual background replacement effects.

[0010] Third, the implementation effect of the virtual background is often related to the running device and the adaptation, and the effects on different devices are different, which results in that the tester needs to adapt a large number of test devices when testing the virtual background business, thereby increasing the cost of effect evaluation.

[0011] Fourth, the virtual background test needs a camera for video acquisition, but the tester may not have so many test devices; considering that the acquisition resolution and frame rate of different cameras are different, and the virtual background effect on different acquisition parameter cameras is also different, the tester needs to adapt it, and there may be a lack of such camera hardware.

[0012] Fifth, the virtual background test is mostly through the test list simulation by the R&D and test personnel, and the image effect is scored according to the subjective consciousness of the person, but the whole process is still subjective, and at the same time, there is no intuitive comparison between the picture after virtual background replacement and the actual picture.

[0013] Sixth, the effect of the virtual background often affects the core competitiveness of the product, at this time, we need to further compare the effect with the competitive product, but if the effect of the competitive product is also verified, it is undoubtedly a large time cost investment. SUMMARY

[0014] The technical problem to be solved by the present application is to provide an automatic evaluation method and terminal for virtual background, which effectively improves the virtual background effect evaluation efficiency.

[0015] In order to solve the above technical problems, the technical scheme adopted by the present application is:

[0016] An automatic evaluation method for virtual background, comprising the steps of:

[0017] S1, the background management module obtains original data and virtual background after the background virtualization of the to-be-evaluated data, obtains the evaluation data pair, configures a plurality of frame rates for the evaluation data pair, and sends the evaluation task to the terminal management module;

[0018] S2, the terminal management module obtains the evaluation data pair and the device list for evaluation from the background management module according to the evaluation task;

[0019] S3, the terminal management module starts the corresponding terminal device to evaluate the evaluation data pair based on the image similarity algorithm according to the device list.

[0020] In order to solve the above technical problems, another technical solution adopted by the present application is:

[0021] An automatic evaluation terminal for virtual background comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the automatic evaluation method for virtual background.

[0022] The present application has the advantages that an automatic evaluation method and terminal for virtual background are provided, the data pair for virtual background effect evaluation and the terminal device for evaluation are obtained by the background management module, the shooting effect of different frame rate cameras is simulated by configuring multiple frame rates for the data pair to adapt to different terminal devices for background effect evaluation, and finally the terminal device compares the background blurred evaluation data in the data pair with the original data based on the image similarity algorithm, so that the evaluation result is accurate, the whole evaluation process only needs to be controlled by the background management module and the evaluation task is sent to the corresponding terminal device for automatic virtual background effect evaluation, which is convenient and fast, and effectively improves the virtual background effect evaluation efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 The whole flow chart of the automatic evaluation method for virtual background of the embodiment of the present application;

[0024] Figure 2 The system service chart of the automatic evaluation method for virtual background of the embodiment of the present application;

[0025] Figure 3 The flow chart of the data management sub-module in the automatic evaluation method for virtual background of the embodiment of the present application;

[0026] Figure 4 The interaction chart of the terminal device and the device management sub-module in the automatic evaluation method for virtual background of the embodiment of the present application;

[0027] Figure 5 The flow chart of the task management sub-module in the automatic evaluation method for virtual background of the embodiment of the present application;

[0028] Figure 6 A flow chart of a data statistics sub-module in a virtual background automatic evaluation method of an embodiment of the present application;

[0029] Figure 7 A flow chart of a terminal scheduling sub-module in a virtual background automatic evaluation method of an embodiment of the present application;

[0030] Figure 8 A flow chart of a test execution sub-module in a virtual background automatic evaluation method of an embodiment of the present application;

[0031] Figure 9 A schematic diagram of an original image in an embodiment of the present application;

[0032] Figure 10 A schematic diagram of a background virtualization in an embodiment of the present application;

[0033] Figure 11 Another schematic diagram of a background virtualization in an embodiment of the present application;

[0034] Figure 12 A structural schematic diagram of a virtual background automatic evaluation terminal of an embodiment of the present application.

[0035] Label explanation:

[0036] 1. A virtual background automatic evaluation terminal; 2. A memory; 3. A processor. DETAILED DESCRIPTION

[0037] To make the technical content, the achieved purposes and effects of the present application clear, the following will be described in detail in combination with the embodiments and the accompanying drawings.

[0038] Please refer to Figures 1 to 11 A virtual background automatic evaluation method, comprising the steps of:

[0039] S1. A background management module acquires original data and to-be-evaluated data after background virtualization of the original data, obtains an evaluation data pair, configures multiple frame rates for the evaluation data pair, and issues an evaluation task to a terminal management module.

[0040] S2. The terminal management module acquires the evaluation data pair and a device list for evaluation from the background management module according to the evaluation task.

[0041] S3. The terminal management module starts a corresponding terminal device to perform virtual background effect evaluation on the evaluation data pair based on an image similarity algorithm according to the device list.

[0042] From the above description, the beneficial effects of the present application are: to provide an automatic evaluation method of virtual background, the background management module is used to obtain the data pair which needs to be evaluated and select the terminal equipment for evaluation, and the shooting effect of different frame rate cameras is simulated by configuring multiple frame rates for the data pair to adapt to different terminal equipment for background effect evaluation, and finally the terminal equipment compares the background virtualization data to be evaluated and the original data in the data pair based on the image similarity algorithm, so as to realize the accuracy of the evaluation result, the whole evaluation process only needs the test personnel to control the group through the background management module and issue the evaluation task to the corresponding terminal equipment for automatic virtual background effect evaluation, which is convenient and fast, and effectively improves the virtual background effect evaluation efficiency.

[0043] Further, the background management module comprises a data management submodule and a task management submodule;

[0044] The step S1 is specifically:

[0045] S11, the data management submodule batch acquires the original data and the background virtualization data to be evaluated imported by the user, and specifies the resolution size after import, the original data and the data to be evaluated are pictures or videos;

[0046] S12, the data management submodule judges whether the original resolution of the original data and the data to be evaluated is consistent with the specified resolution, if yes, it is directly stored as an evaluation data pair, otherwise, the original data and the data to be evaluated are marked with data information that does not meet the resolution, and then they are stored as the evaluation data pair;

[0047] S13, the task management submodule creates an evaluation task, configures the evaluation data pair with specified resolution and the terminal equipment for evaluation according to the evaluation requirement, and configures the required frame rate for the evaluation data pair and selects the required evaluation application for the terminal equipment;

[0048] S14, the task management submodule issues the evaluation task to the terminal management module.

[0049] From the above description, the background management module is the core module responsible for the whole system business management, the data management submodule included therein can acquire and store the data to be evaluated, and can also specify the resolution of the acquired data for subsequent evaluation; in addition, the background management module further comprises a task management submodule, which can be used for creating, requirement configuration and task issuing of the evaluation task, so that the terminal management module executes the corresponding evaluation instruction according to the evaluation task, and effectively improves the virtual background effect evaluation efficiency.

[0050] Further, the step S12 further comprises:

[0051] convert the original resolution of the evaluation data pair with the data information not meeting the resolution into a specified resolution, and modify the data information not meeting the resolution into data information after conversion of the resolution.

[0052] As can be seen from the above description, in order to further improve the efficiency of subsequent virtual background effect evaluation, the original resolution of the data pair not meeting the specified resolution can also be converted into the specified resolution directly in the data management submodule.

[0053] Further, the background management module further comprises a device management submodule;

[0054] The step S2 further comprises:

[0055] The device management submodule records the MAC address and IP address of each terminal device, and registers a unique login number for each terminal device, and then sends the login number to the terminal management module;

[0056] When a new terminal device is added, the registration request of the new terminal device is obtained, the MAC address and IP address of the new terminal device are stored, and the new login number is sent to the terminal management module.

[0057] As can be seen from the above description, the background management module further comprises a device management submodule, which can be used for unified management of each terminal device under the terminal management module, i.e. recording the MAC address and IP address of each terminal device, and configuring a unique login number for each terminal device, so that the connection between the background management module and the specified terminal device can be quickly established according to the mapping relationship between the MAC address, IP address and login number in the subsequent virtual background effect evaluation work.

[0058] Further, the terminal management module comprises a terminal scheduling submodule and a test execution submodule;

[0059] The step S2 specifically comprises:

[0060] The terminal scheduling submodule receives the evaluation task, parses the content of the evaluation task to generate an execution instruction, and then sends the execution instruction to the test execution submodule;

[0061] The execution instruction comprises obtaining the required evaluation data pair, obtaining a device list, establishing a long link between the terminal device performing evaluation and the background management system, and deploying the required evaluation application in the terminal device performing evaluation.

[0062] As can be known from the above description, the terminal management module, as an execution end for being responsible for a specific evaluation task, comprises a terminal scheduling submodule and a test execution submodule, wherein the terminal scheduling submodule is mainly responsible for receiving an evaluation task issued by the background management module and parsing the evaluation task into an execution instruction for the test execution submodule to perform corresponding execution, and the two have clear division of labor, thereby further improving the evaluation efficiency of the virtual background effect.

[0063] Further, the step S3 specifically comprises:

[0064] S31, the test execution submodule receives the execution instruction, obtains the required evaluation data pair from the background management module, and generates a device list according to the login number of the terminal device for evaluation;

[0065] S32, the test execution submodule starts the corresponding terminal device according to the device list, and initiates a request for establishing a long link to the background management module according to the login number of the corresponding terminal device;

[0066] S33, the device management submodule finds the corresponding MAC address and IP address according to the login number, establishes a long link with the corresponding terminal device, and activates the current state of the corresponding terminal device as an online state;

[0067] S34, the test execution submodule deploys the evaluation application for the terminal device with the current state as an online state, and runs the evaluation application to perform virtual background effect evaluation on the obtained evaluation data pair based on an image similarity algorithm.

[0068] Further, in the step S34, the evaluation application is deployed for the online terminal device, specifically comprising:

[0069] parsing a download address of the evaluation application, if the download address exists in a download record in the local of the online terminal device, directly obtaining the evaluation application from a local application list of the online terminal device;

[0070] otherwise, downloading the evaluation application through the download address and installing the evaluation application in the local of the online terminal device;

[0071] Before the step S34, the evaluation application is run, further comprising:

[0072] calling an application automation enabler in the local of the online terminal device to initialize the evaluation application.

[0073] From the above description, it can be seen that, when the online terminal device is running the specified evaluation application for virtual background effect evaluation of the to-be-evaluated data pair, in order to avoid the problem that the evaluation application does not exist and causes the evaluation work to fail, it is necessary to first judge whether the specified evaluation application already exists in the online terminal device, if yes, directly call and run, otherwise, need to be downloaded and installed before running, and after installation, it is stored in the local file of the online terminal device, and when the device is started again for running of the specified evaluation application, it can be directly called, which effectively improves the virtual background effect evaluation efficiency.

[0074] Further, the step S34 of evaluating the obtained evaluation data pair based on the image similarity algorithm is specifically:

[0075] Judging the type of the to-be-evaluated data pair, if it is a video, cutting each frame of picture according to the frame rate, if it is a picture, no cutting processing is needed;

[0076] Calling the OBS virtual camera plug-in of the terminal device to read each pair of pictures in the to-be-evaluated data pair in turn according to the frame rate configured in the evaluation task, and introducing the uniform error algorithm and the peak signal-to-noise ratio algorithm to evaluate the similarity of each pair of pictures.

[0077] From the above description, it can be seen that, by simulating the virtual camera push stream function, the consistency of each evaluation can be ensured, and the evaluation result error caused by changes of the to-be-evaluated data pair or the online terminal device for evaluation can be avoided.

[0078] Further, the background management module further comprises a data statistics submodule;

[0079] The step S3 further comprises:

[0080] S4, the terminal management submodule uploads the evaluation result to the data statistics submodule, the data statistics submodule visually displays the evaluation result, collects the subjective score of the customer on the visual display of the evaluation result, and calculates the final score of the background virtualization of the evaluation data pair by comprehensively considering the evaluation result and the subjective score.

[0081] From the above description, it can be seen that the background management module further comprises a data statistics submodule, which stores and visually displays the evaluation result, and can report and statistics to quickly quantify the optimization improvement of the virtual background effect evaluation of each version and the performance occupation of each version terminal device in the virtual background effect evaluation, so as to provide basis for whether the product needs further optimization in performance.

[0082] Please refer to Figure 12An automatic evaluation terminal of a virtual background comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps in the automatic evaluation method of a virtual background when executing the computer program.

[0083] From the above description, the beneficial effects of the present application are that based on the same technical concept, in combination with the above-mentioned automatic evaluation method of a virtual background, an automatic evaluation terminal of a virtual background is provided, the background effect evaluation data pair to be evaluated is obtained by a background management module, and the terminal device to be evaluated is selected, and the shooting effect of a camera with different frame rates is simulated by configuring multiple frame rates for the data pair to adapt to different terminal devices for background effect evaluation, and finally the terminal device compares the background blurred evaluation data in the data pair with the original data based on an image similarity algorithm, so that the evaluation result is accurate, the entire evaluation process only needs to be controlled by the background management module by the test personnel and the evaluation task is issued to the corresponding terminal device for automatic virtual background effect evaluation, which is convenient and fast, and effectively improves the virtual background effect evaluation efficiency.

[0084] The present application provides an automatic evaluation method and terminal of a virtual background, mainly applied to the scene of evaluating the virtual background replacement effect in the Internet, which will be specifically described below in combination with specific embodiments:

[0085] Please refer to Figure 1 The embodiment one of the present application is:

[0086] An automatic evaluation method of a virtual background, as shown in Figure 1 The steps include:

[0087] S1, the background management module obtains the original data and the evaluation data after the background virtualization, obtains the evaluation data pair, configures multiple frame rates for the evaluation data pair, and issues the evaluation task to the terminal management module.

[0088] S2, the terminal management module obtains the required evaluation data pair and the device list for evaluation from the background management module according to the evaluation task.

[0089] S3, the terminal management module starts the corresponding terminal device based on the image similarity algorithm to evaluate the virtual background effect of the evaluation data pair according to the device list.

[0090] That is, in the embodiment, the background management module first acquires the data pair to be evaluated and selects the terminal device to be evaluated, and simulates the shooting effect of the camera with different frame rates by configuring multiple frame rates for the data pair to adapt to the background effect evaluation of different terminal devices, and finally the terminal device compares the original data and the background blurred data to be evaluated in the data pair based on the image similarity algorithm to realize the accuracy of the evaluation result. The whole evaluation process only needs the test personnel to control the background management module and issue evaluation tasks to the corresponding terminal device for automatic virtual background effect evaluation, which is convenient and fast, and effectively improves the virtual background effect evaluation efficiency.

[0091] Please refer to Figures 2 to 11 , the second embodiment of the present application is:

[0092] An automatic evaluation method of virtual background, based on the first embodiment, in the embodiment, the background management module is the core module responsible for the business management of the whole system, as shown in Figure 2 , the background management module is divided into four small modules, including data management submodule, task management submodule, device management submodule and data statistics submodule.

[0093] In the embodiment, as shown in Figure 3 , step S1 is specifically:

[0094] S11, the data management submodule batch acquires the original data and the background virtualized data to be evaluated uploaded by the user, and specifies the resolution size after uploading, the original data and the data to be evaluated are pictures or videos.

[0095] At the same time, in the embodiment, the data management submodule also supports providing corresponding remark information for the original data and the background blurred data to be evaluated. When the user uploads pictures or videos, the corresponding resolution information will be displayed by default, which facilitates the subsequent test evaluation and quickly selects the data the user wants.

[0096] S12, the data management submodule judges whether the original resolution of the original data and the data to be evaluated is consistent with the specified resolution, if yes, it is directly stored as an evaluation data pair, otherwise, the data information of the original data and the data to be evaluated which do not meet the resolution is marked, and then they are stored as an evaluation data pair, and after the data is imported, the evaluation data pair and the resolution information details of the evaluation data pair which do not meet the resolution are informed.

[0097] At the same time, in the embodiment, the resolution conversion process is also provided, that is, step S12 further includes:

[0098] The original resolution of the evaluation data pair with data information not meeting the resolution is converted to a specified resolution, and the data information not meeting the resolution is modified to data information after the conversion.

[0099] That is, the user can force the evaluation data pair not meeting the resolution to be converted to data of the specified resolution, so as to further improve the efficiency of subsequent virtual background effect evaluation. For the data pair not meeting the specified resolution, the original resolution thereof can also be directly converted to the specified resolution in the data management submodule, and after the resolution conversion is completed, the data is stored in the corresponding batch of data pair sets and marked as converted data.

[0100] Meanwhile, the embodiment also provides a behavior of deleting the evaluation data pair, for maintaining the data. In addition, the background management module also provides a general configuration module for maintaining the evaluation application information, which includes the application name of the evaluation application, the download address of the evaluation application, and the automatic enablement program download address of the evaluation application, etc.

[0101] In the embodiment, the task management submodule is responsible for the creation, configuration and issuance of the task, as shown in Figure 5 The process of the task management submodule is as follows:

[0102] S13, the task management submodule creates an evaluation task, selects one or more terminal devices for the evaluation task according to the task requirement, and configures the evaluation data pair of the specified resolution. The process can select by searching the remark information of the evaluation data pair or screening the resolution, and then configures the frame rate required by the camera simulation for the evaluation data pair. Finally, the terminal device selects the required evaluation application. The evaluation application name can be selected singly or multiple, for example, an internal self-developed virtual background application software is selected, and a competitive application software is also selected.

[0103] S14, when the configuration is completed, the task management submodule issues the evaluation task to the terminal management submodule for further implementation.

[0104] It is worth noting that in the embodiment, the device management submodule of the background management module is responsible for managing the terminal device. In the embodiment, as shown in Figure 4 Before step S2, it also includes:

[0105] The device management submodule records the MAC address and IP address of each terminal device, and registers a unique login number for each terminal device, and then sends the login number to the terminal management module. Subsequently, when the terminal device starts, the starting terminal device will initiate a login request to the device management submodule according to the login number, and the device management submodule will link the MAC address and IP address corresponding to the login number. The connection here is a Socket long connection, which can ensure the two-way instant communication between the device and the background during the evaluation process. When the terminal device and the background Socket link are online, it will be identified as online, and when the link is disconnected, it will be identified as offline.

[0106] When a new terminal device is added, the device management submodule will obtain the registration request of the new terminal device, that is, the new terminal device will initiate a registration request to the device management submodule with its own MAC address and IP address. When the device management submodule stores the MAC address and IP address of the new terminal device and determines that the MAC address and IP address are new, that is, the terminal device is a new device, it will register a new login number and send the new login number to the terminal management module to return to the new terminal device.

[0107] For example Figure 2 As shown in the figure, the terminal management module is the core module responsible for specific test execution, which is divided into two small modules, including a terminal scheduling submodule and a test execution submodule.

[0108] In this embodiment, as shown in the figure, step S2 is specifically: Figure 7

[0109] The terminal scheduling submodule receives the evaluation task and parses the content of the evaluation task to generate an execution instruction, and then sends it to the test execution submodule.

[0110] The execution instruction includes obtaining the required evaluation data pair, obtaining the device list, establishing a long link between the terminal device performing evaluation and the background management system, and deploying the required evaluation application in the terminal device performing evaluation.

[0111] That is, the terminal management module serves as an execution end responsible for specific evaluation tasks, including a terminal scheduling submodule and a test execution submodule. The terminal scheduling submodule is mainly responsible for receiving the evaluation task issued by the background management module, and parsing it into an execution instruction for the test execution submodule to perform accordingly. The division of labor between the two is clear, which further improves the evaluation efficiency of the virtual background effect.

[0112] First, as known from the above, each terminal device will initiate registration to the device management submodule of the background management module when it is deployed, mainly to report its device information, including the MAC address and IP address.

[0113] ​And the terminal device selected for evaluation test in the subsequent test execution submodule instruction execution, will be launched to the device management submodule establishment Socket long link request, to maintain the terminal device and the background between the communication, that is, the communication between the client and the server, so that even if the subsequent terminal device deployment location and the background service end not in a LAN, the server can get some real-time data updates to the terminal device, that is, in the embodiment, step S3 is specifically:

[0114] Test execution submodule is the core test evaluation implementation module of the embodiment, which can execute according to the evaluation task issued by the background, and report the test data and performance data of the evaluation process to the background management module, such as Figure 8 As shown in the figure, the step process is as follows:

[0115] S31, the test execution submodule receives the execution instruction, obtains the required evaluation data pair from the background management module, and generates a device list according to the login number of the terminal device for evaluation.

[0116] S32, the test execution submodule starts the corresponding terminal device according to the device list, and initiates a long link request to the background management module according to the login number of the corresponding terminal device.

[0117] S33, the device management submodule finds the corresponding MAC address and IP address according to the login number, establishes a long link with the corresponding terminal device, and activates the current state of the corresponding terminal device to online state.

[0118] S34, the test execution submodule deploys evaluation application for the terminal device with online state, and runs the evaluation application to evaluate the obtained evaluation data pair based on image similarity algorithm.

[0119] Among them, the evaluation application is deployed for the online terminal device, specifically:

[0120] The download address of the evaluation application is parsed. If the download address exists in the local download record of the online terminal device, the evaluation application is directly obtained from the local application list of the online terminal device; otherwise, the evaluation application is downloaded through the download address and installed in the local of the online terminal device. That is, after downloading, the downloaded application and the corresponding update time and download address will be updated in the local configuration file, which is convenient for next evaluation test, and also used for checking whether further update is needed. The same method can also be used for obtaining the evaluation data pair.

[0121] Once the evaluation application is ready, the application automation launcher will be invoked to initialize it. The main function of this launcher is to enable the test application, simulate opening its virtual background function, maximize its page display, and select the OBS camera application. In other words, before running the specified evaluation application to evaluate the virtual background effect of the data pair on the online terminal device, to avoid evaluation failure due to the application's absence, it needs to first determine if the specified evaluation application already exists on the online terminal device. If it does, it will be run directly; otherwise, it needs to be downloaded and installed before running. After installation, the application is stored locally on the online terminal device. Subsequent restarts of the device to run the specified evaluation application will then allow for direct invocation, effectively improving the efficiency of virtual background effect evaluation.

[0122] In this embodiment, step S34 involves evaluating the virtual background effect of the acquired evaluation data pairs based on an image similarity algorithm, specifically as follows:

[0123] The system determines the type of the data pair to be evaluated. If it's video, it's segmented into individual frames based on the frame rate; otherwise, no segmentation is needed. Then, the OBS virtual camera plugin on the terminal device is invoked to sequentially read each pair of images from the data pair to be evaluated at the frame rate configured in the evaluation task. An average error distribution algorithm and a peak signal-to-noise ratio algorithm are used to evaluate the similarity of each pair of images. This way, regardless of the video's frame rate, the actual output effect can be based on our custom frame rate.

[0124] Even when the data source is determined to be an image, the number of frames can still be pushed in a loop within one second according to the frame rate set in the backend.

[0125] In other words, by simulating streaming through a virtual camera, the consistency of each evaluation can be guaranteed, and the evaluation results will not be affected by changes in the data to be evaluated or the online terminal device used for evaluation.

[0126] During the evaluation process, local screen recording will be enabled to facilitate troubleshooting and pinpointing issues during subsequent testing. Simultaneously, a recording will be taken every second. Figure 1 The system displays a virtual background and records the current system performance and the performance usage of the test application, including CPU, GPU, memory, and bandwidth usage.

[0127] In this embodiment, based on the effect picture of the virtual background and the original input picture, a similarity evaluation is performed, and the mean squared error (MSE) and the peak signal-to-noise ratio (PSNR) are used to evaluate the similarity of the two pictures. Of course, under the effect of the virtual background replacement, the picture similarity is difficult to approach 100%, but the score is higher if the pixel difference between the corresponding area of the original picture is smaller. Figure 10 and Figure 11 For example, if the area pointed by the arrow can be replaced by the original Figure 1 picture, the score is higher because the pixel difference between the corresponding area of the original picture is smaller. In the version optimization, the increase of the score objectively represents the improvement of the model effect.

[0128] In addition, in this embodiment, after the effect evaluation of a set of evaluation data pairs is completed, the related program is closed, and the evaluation process record and the evaluation result are reported to the data statistics submodule of the background management module. In addition, during the evaluation process, if the terminal device detects that the virtual background fails to be enabled or the black screen scene occurs, an exception request is directly triggered to the data statistics submodule, the related early warning push is triggered by the background, and the current evaluation is ended.

[0129] That is, in this embodiment, as shown in Figure 6 , after step S3, the following steps are further included:

[0130] S4, the terminal management submodule uploads the evaluation result to the data statistics submodule, the data statistics submodule visually displays the evaluation result, collects the subjective score of the customer on the visual display of the evaluation result, and calculates the final score of the background virtualization of the evaluation data pair by comprehensively evaluating the evaluation result and the subjective score.

[0131] That is, the data statistics submodule is responsible for the data reporting, receiving, and statistical display of the evaluation process of the terminal management module, and supports the push management configuration in the related exception. Specifically:

[0132] First, the data statistics submodule provides a set of interfaces for the terminal management module to report, including an exception feedback interface and an evaluation report uploading interface.

[0133] The exception feedback interface cooperates with the push configuration function of the data statistics submodule, and when the related exception occurs, the early warning notification is directly performed according to the push configuration channel. For example, during the virtual background effect evaluation process of the terminal device, the black screen or program crash occurs, at which time the background management module is triggered to notify, and the notification is performed according to the configured notification channel (such as 99U, email, etc.).

[0134] The evaluation report upload interface is used to receive the evaluation results reported by the terminal management module. When the terminal management module completes the evaluation of the corresponding terminal device, it uploads the evaluation results to the background.

[0135] When the evaluation report is uploaded to the background, the data statistics submodule displays the objective evaluation results of all terminal devices under the current task ID based on the task ID. The user can click to view the evaluation results on a terminal device, and clearly see a comparison preview of each picture before and after the virtual background conversion, as well as the current scoring results and the current terminal device performance occupation.

[0136] Each evaluation result can be manually scored and shared. The system supports issuing objective scoring links in the form of a questionnaire. When the user obtains the corresponding scoring link, they can intuitively see the effect before and after the virtual background, and then perform corresponding subjective scoring.

[0137] When all users have completed scoring, the data statistics submodule automatically calculates the highest score, the lowest score, the average score, etc. More preferably, the data statistics submodule supports a comparison function that can pull the test results of the same test data and the same test device from history. The two results are displayed side by side on a report to show the results. That is, the user can easily see the original picture, the replacement effect picture of the previous version of the virtual background, and the replacement effect picture of the latest version of the virtual background, and compare the subjective and objective scoring results and performance occupation of the two virtual background replacement effects.

[0138] The following provides several specific application scenarios to illustrate the effectiveness of the virtual background automatic evaluation method of this embodiment:

[0139] Scenario One:

[0140] Test student A is responsible for testing a live streaming project. To improve the live streaming effect and enhance the competitiveness of the product, the product proposes a virtual background function module. A needs to evaluate the implementation effect of the virtual background. A wants to better evaluate the replacement effect of the virtual background, and fully lists various combinations of people, objects, and backgrounds. After development and testing, A spends time testing a round in front of the camera, especially in the multi-person entry scene, and even takes pictures with colleagues for verification. After A returns the test results, the developer quickly finds that the effect is worse than expected, adjusts the model and retests, and A has to verify the combination scenarios again. Due to various reasons, A has submitted multiple rounds of testing, and A's enthusiasm has gradually waned from the initial passion.

[0141] Through the scheme of the embodiment, the small A only needs to record a test scene once, and collect each scene to be shot. Thereafter, the small A only needs to import the picture set shot by the small A through the scheme, and then distribute the picture set to the test machine of the small A. The small A can obtain a test result, and can intuitively see the test result of the virtual background on the data statistics submodule. In the subsequent development and submission of the test record, the small A only needs to update the latest test package address on the data management submodule, and then re-distribute the test task to quickly obtain the latest test result of the virtual background.

[0142] Scene two:

[0143] The small A responsible for the module test of the virtual background needs to evaluate the implementation effect of the virtual background. In the process of the small A performing different version iteration tests on the test list of the virtual background, it is found that the same person enters the mirror, and sometimes just a slight adjustment of the action position can produce different replacement effects on the virtual background. In order to try to keep consistent with the previous position and action, the small A will be very tired in the process of "posing". In addition, it is found that the test is submitted, and the clothes color of the small A is different from the background, which also has a great effect on the test result. At the same time, when the development problem is submitted, the development will complain that the test result is different. In fact, the essence is that the test case is changed, and in addition to the clothes color, the light is also different. For his program, the input picture is already different, and the result is different. It is normal. At this time, the small A also needs to take the old package and compare the "posing" of the scene, which is a relatively cumbersome process.

[0144] Through the scheme of the embodiment, as long as the small A imports the test data in the background, the test data is fixed regardless of how many versions are submitted by the development, which solves the time investment of the small A in the so-called posing each time. At the same time, if new effects need to be supplemented in the new scene, the small A can also add new test data to regress the related version, and use it to quantify the difference between the past and the present.

[0145] Scene three:

[0146] Test classmate A is responsible for the module test of virtual background. During the test process, A finds that the test effect of his own work machine is relatively ideal, but when he uses another notebook, the whole system is very slow, and the virtual background effect is very blurred and poor. During the communication with the developer, it is found that the virtual background model still has many adaptation problems and performance requirements for machine types, especially more than that, the subsequent development will also use GPU power model according to the type of graphics card. Therefore, A not only needs to test the work machine, but also needs to adapt to different manufacturers' notebooks. He also needs to consider the type of graphics card to facilitate the implementation of the test and development logic, whether to use GPU first and then CPU. However, A finds it very troublesome to simply evaluate the virtual background effect of a single device. He needs to invest a lot of time to test many devices at the same time. At the same time, A finds that the interface of his colleague's device is directly stuck. The reason is that this computer uses a high-definition camera. Later, the developer informed that the virtual background cannot handle high-resolution and high-frame-rate input sources, but A does not have such a camera hardware.

[0147] Through the scheme of this embodiment, A only needs to install the terminal management submodule installation package for his colleagues, and can select these devices in the background in batches. When the colleagues are idle, A can issue test tasks to complete many adaptation tests. At the same time, A can set the corresponding configuration according to the resolution and frame rate of the common camera in the industry to simulate the input effect of different cameras. Thus, the problems of resource shortage and adaptation test workload of test computer devices and camera devices are solved.

[0148] Scenario four:

[0149] Test classmate A is responsible for the module test of virtual background. For the replacement effect of virtual background, A often can only give his subjective evaluation. When the person in charge asks about the replacement effect of virtual background, A's feedback is more about the phenomenon existing in the current virtual background replacement process. For example Figure 8 is the original picture, Figure 9 is the virtual background replacement effect. A can only feedback that there is a partial scene where the character will be out of the model, but cannot quantify the influence size. Especially in the process of version iteration, A has difficulty in describing the quantity of optimization.

[0150] Through the scheme of the embodiment, as long as small A issues a task through the manner provided by the system, performs testing and execution, and records an intuitive similarity comparison of each original picture and virtual background replacement picture in the process, based on this, when multiple versions are iterated, the similarity difference between two versions can be quantified, which is used as reference data for effect improvement. Figures 9 to 11 As shown in the figure, Figure 9 is an original picture, Figure 10 is a virtual background replacement effect of a first version. From the intuitive similarity of the picture, it can be clearly known that Figure 10 is about 46% similar to Figure 9 However, the entire hand area has not been restored. Through optimization of the program, Figure 11 is about 8% similar to Figure 9 It is obvious that the hand area is restored. That is, the restoration degree of the first version is 46%, and the restoration degree of the second version is 56%, which can be used as reference data for restoration of a person's image.

[0151] Scene five:

[0152] Test student small A is responsible for virtual background module testing. Through iteration of multiple versions, small A feels that the product of small A basically meets the release standard. After release online, small A receives many criticisms, and many users feel that the virtual background replacement effect of the product of small A is poor, and the replacement effect of the product of small A is poor in many scenes compared with many competitors online. Therefore, small A urgently takes some competitors to compare and distinguish the effect of the product of small A. After that, when the core competitors release updates, the leader of small A will ask small A to pay attention to the optimization effect of the competitors. For small A, the product of small A is tested, and small A also needs to test other products from time to time, and feels that the time investment is much.

[0153] Through the scheme of the embodiment, small A only needs to configure a to-be-tested application in the background, including a download address of a corresponding competitor application and an enabling test application of the competitor application, so that the internal application and the competitor application can be simultaneously selected when a task is issued. During the entire testing process, the original picture, the virtual background replacement effect of the internal application, and the virtual background replacement effect of the competitor can be visually seen in the background, and the first objective score of both sides can be seen. At the same time, a score result of the internal application and the competitor can be collected through a questionnaire, so that the subjective and objective score difference between the product of small A and the competitors can be collected.

[0154] Please refer to Figure 12 , the third embodiment of the present application is:

[0155] An automatic evaluation terminal 1 of a virtual background comprises a memory 2, a processor 3, and a computer program stored in the memory 2 and executable on the processor 3, and the processor 3 completes the steps in the automatic evaluation method of the virtual background in any one of the above-mentioned embodiments when executing the computer program.

[0156] In summary, the automatic evaluation method and terminal of a virtual background provided by the present application have the following beneficial effects:

[0157] 1. By the virtual background testing method provided by the present application, testers can easily test the virtual background through the test terminal of the background group control.

[0158] 2. By the virtual background testing method provided by the present application, testers can simulate the configuration of different camera resolutions and frame rates.

[0159] 3. By the virtual background testing method provided by the present application, the internal and external network environments of the test client can be ignored, and even if the test equipment is not in the same local area network, the corresponding task can be issued for testing.

[0160] 4. By the report statistics provided by the present application, the optimization and improvement effect of each version of the virtual background can be quickly quantified.

[0161] 5. By the report statistics provided by the present application, the performance occupation of each version of the virtual background can be quickly quantified, which provides a basis for whether the product needs further optimization in terms of performance.

[0162] 6. By the camera simulation push function provided by the present application, the consistency of test data can be ensured, and the test result error caused by test data or test equipment change can be avoided.

[0163] 7. By the subjective scoring questionnaire method provided by the present application, the test and user experience results can be quickly summarized together, and the results are more authoritative and quantified.

[0164] 8. By the process exception feedback provided by the present application, even if the process has related exceptions, relevant personnel can troubleshoot related errors and exceptions in the first time through early warning.

[0165] The above-mentioned embodiments are only examples of the present application, and do not limit the patent scope of the present application, and any equivalent transformation or direct or indirect application in related technical fields based on the content of the present application specification and drawings is also included in the patent protection scope of the present application.

Claims

1. A method for automated evaluation of a virtual background, characterized in that, The method comprises the steps of: S1, the background management module obtains original data and its background virtualization after the evaluation data, obtains the evaluation data pair, and configures a plurality of frame rates for the evaluation data pair, and issues an evaluation task to the terminal management module; S2, the terminal management module obtains the required evaluation data pair and the device list for evaluation from the background management module according to the evaluation task; S3, the terminal management module starts the corresponding terminal device based on the image similarity algorithm to evaluate the evaluation data pair according to the device list; The background management module comprises a data management submodule and a task management submodule; The step S1 is specifically: S11, the data management submodule batch obtains user-defined imported original data and its background virtualization after the evaluation data, and specifies the resolution size after import, and the original data and the evaluation data are pictures or videos; S12, the data management submodule judges whether the original resolution of the original data and the evaluation data is consistent with the specified resolution, if yes, it is directly stored as an evaluation data pair, otherwise, the original data and the evaluation data are marked with data information that does not meet the resolution, and then they are stored as the evaluation data pair; S13, the task management submodule creates an evaluation task, configures the evaluation data pair with specified resolution and terminal device for evaluation according to the evaluation requirement, configures the required frame rate for the evaluation data pair, and selects the required evaluation application for the terminal device; S14, the task management submodule issues the evaluation task to the terminal management module; The step S3 is specifically: S31, the test execution submodule receives an execution instruction, obtains the required evaluation data pair from the background management module, and generates a device list according to the login number of the terminal device for evaluation; S32, the test execution submodule starts the corresponding terminal device according to the device list, and initiates a request to establish a long link to the background management module according to the login number of the corresponding terminal device; S33, the device management submodule finds the corresponding MAC address and IP address according to the login number, establishes a long link with the corresponding terminal device, and activates the current state of the corresponding terminal device to online state; S34, the test execution submodule deploys the evaluation application for the terminal device with online state, and runs the evaluation application to evaluate the evaluation data pair based on the image similarity algorithm.

2. The method of claim 1, wherein, In the step S12, the original resolution of the evaluation data pair with the data information that does not meet the resolution is converted to the specified resolution, and the data information that does not meet the resolution is modified to the data information after conversion. The background management module further comprises a device management submodule; 3. The method of claim 1, wherein, Before the step S2, it further comprises: The device management submodule records the MAC address and IP address of each terminal device, registers a unique login number for each terminal device, and sends the login number to the terminal management module; ​ When a new terminal device is added, a registration request of the new terminal device is acquired, the MAC address and IP address of the new terminal device are stored, and a new login number is sent to the terminal management module.

4. The method of automated evaluation of a virtual background of claim 3, wherein, The terminal management module comprises a terminal scheduling submodule and a test execution submodule. The step S2 is specifically: The terminal scheduling submodule receives the evaluation task, parses the content of the evaluation task to generate an execution instruction, and then sends the execution instruction to the test execution submodule. The execution instruction comprises acquiring the required evaluation data pair, acquiring a device list, establishing a long link between the terminal device for evaluation and the background management module, and deploying the required evaluation application in the terminal device for evaluation.

5. The method of claim 1, wherein, In the step S34, the evaluation application is deployed in the online terminal device, specifically: The download address of the evaluation application is parsed, if the download address exists in the download record in the local of the online terminal device, the evaluation application is directly acquired from the local application list of the online terminal device; Otherwise, the evaluation application is downloaded through the download address and installed in the local of the online terminal device; Before running the evaluation application in the step S34, the following steps are further included: An application automation enabler in the local of the online terminal device is called to initialize the evaluation application.

6. The method of claim 1, wherein, In the step S34, the evaluation data pair acquired is evaluated based on an image similarity algorithm, specifically: The type of the data pair to be evaluated is judged, if it is a video, it is cut into each frame of picture according to the frame rate, if it is a picture, no cutting processing is needed; The OBS virtual camera plug-in of the terminal device is called to read each pair of pictures in the data pair to be evaluated in turn according to the frame rate configured in the evaluation task, and the mean square error algorithm and the peak signal-to-noise ratio algorithm are introduced to evaluate the similarity of each pair of pictures.

7. The method of claim 1, wherein, The background management module further comprises a data statistics submodule. After the step S3, the following steps are further included: S4, the terminal management module uploads the evaluation result to the data statistics submodule, the data statistics submodule visually displays the evaluation result, collects the subjective score of the customer on the visual display of the evaluation result, and calculates the final score of the background virtualization of the evaluation data pair by comprehensively considering the evaluation result and the subjective score.

8. An automated evaluation terminal for virtual backgrounds, characterized by The computer program stored in the memory and executable on the processor comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to realize the steps in the virtual background automatic evaluation method in any one of claims 1 to 7.

Images