Interaction information testing method, device, electronic device and storage medium

Abstract

The present disclosure provides an interactive information testing method, which relates to the field of artificial intelligence technology, especially to the technical fields of automated testing, deep learning and augmented reality, and can be applied to scenarios such as smart home and smart customer service. The specific implementation scheme is: perform target detection on multiple images to be tested from the interactive information respectively, and obtain multiple first detection results, wherein the first detection result includes a first detection frame of at least one object to be tested in the image to be tested; determine at least one first partial image based on the first detection result; determine an image comparison result based on at least one first partial image and at least one second partial image, wherein at least one second partial image is determined based on the second detection result, and the second detection result is obtained by performing target detection on a preset image; and determine the test result based on the image comparison result. The present disclosure also provides an interactive information testing device, an electronic device and a storage medium.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to the fields of automated testing, deep learning, virtual reality and augmented reality, and can be applied to scenarios such as smart homes and smart customer service. More specifically, the present disclosure provides an interactive information testing method, device, electronic device and storage medium. Background Art

[0002] With the development of artificial intelligence technology, users have put forward higher requirements for human-computer interaction technology. In various fields, some smart devices (such as smart speakers) can provide multimodal information interaction functions. For example, a smart device can display a virtual image. The virtual image can interact with the user through voice, facial expressions, body movements, etc. Summary of the invention

[0003] The present disclosure provides an interactive information testing method, device, equipment and storage medium.

[0004] According to one aspect of the present disclosure, a method for testing interactive information is provided, the method comprising: performing target detection on multiple images to be tested from interactive information respectively to obtain multiple first detection results, wherein the first detection result includes a first detection frame of at least one object to be tested in the image to be tested; determining at least one first local image based on the first detection result; determining an image comparison result based on the at least one first local image and at least one second local image, wherein the at least one second local image is determined based on the second detection result, and the second detection result is obtained by performing target detection on a preset image; and determining a test result based on the image comparison result.

[0005] According to another aspect of the present disclosure, an interactive information testing device is provided, which includes: a target detection module, which is used to perform target detection on multiple images to be tested from the interactive information respectively, and obtain multiple first detection results, wherein the first detection result includes a first detection frame of at least one object to be tested in the image to be tested; a first determination module, which is used to determine at least one first local image according to the first detection result; a second determination module, which is used to determine an image comparison result according to at least one first local image and at least one second local image, wherein at least one second local image is determined according to the second detection result, and the second detection result is obtained by performing target detection on a preset image; and a third determination module, which is used to determine the test result according to the image comparison result.

[0006] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method provided according to the present disclosure.

[0007] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided. The computer instructions are used to cause a computer to execute the method provided according to the present disclosure.

[0008] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements the method provided according to the present disclosure when executed by a processor.

[0009] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0011] Figure 1 is a schematic diagram of an exemplary system architecture to which an interactive information testing method and apparatus can be applied according to an embodiment of the present disclosure;

[0012] Figure 2 is a flow chart of a method for testing interaction information according to an embodiment of the present disclosure;

[0013] Figure 3 is a flow chart for determining an image comparison result according to another embodiment of the present disclosure;

[0014] Figure 4 is a flow chart for determining an image comparison result according to another embodiment of the present disclosure;

[0015] Figure 5 is a flow chart for determining test results according to another embodiment of the present disclosure;

[0016] Figure 6 is a flow chart for determining test results according to another embodiment of the present disclosure;

[0017] Fig. 7A is a schematic diagram of an image to be tested according to an embodiment of the present disclosure;

[0018] Figure 7B and Figure 7C is a schematic diagram of a first partial image according to an embodiment of the present disclosure;

[0019] Figure 8 is a block diagram of an interactive information testing device according to an embodiment of the present disclosure;

[0020] Fig. 9 is a block diagram of an electronic device to which the interaction information testing method can be applied according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0021] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0022] Smart devices or program products with interactive functions can be tested to determine whether the smart devices can meet the preset requirements. For example, after the hardware or software of the smart device is updated, the smart device can be tested based on manual testing. However, the manual testing method requires high labor costs. In addition, manual testing has poor flexibility. In the process of multimodal information interaction, smart devices can display different virtual images, different image backgrounds, and can also interact with the virtual image's body movements, voice, etc. at the same time. Manual testing is prone to omissions or incorrect test results.

[0023] Figure 1 is a schematic diagram of an exemplary system architecture to which the interactive information testing method and device can be applied according to an embodiment of the present disclosure. It should be noted that: Figure 1 What is shown is merely an example of a system architecture to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0024] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0025] Users can interact with terminal devices 101, 102, 103 to receive or send messages, etc. Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, etc. The display screens of terminal devices 101, 102, 103 can, for example, display virtual images to provide multimodal interaction for users. During the multimodal interaction process, the virtual image can display various facial expressions and body movements, and can also make voice. In addition, during the multimodal interaction process, terminal devices 101, 102, 103 can also display various images or texts on the display screen.

[0026] The server 105 may be a server that provides various services, such as a background management server (only an example) that provides support for websites browsed by users using the terminal devices 101, 102, and 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0027] It should be noted that the interactive information testing method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the interactive information testing device provided in the embodiment of the present disclosure can generally be set in the server 105. The interactive information testing method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the interactive information testing method device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.

[0028] In some embodiments, an object detection model can be trained to perform a mutual information test.

[0029] For example, the target detection model may be a YOLO (You Only Look Once) v5 model.

[0030] For example, a plurality of sample images may be acquired to train the target detection model. For another example, images on a display screen of the smart device described above may be acquired as sample images.

[0031] For example, the sample image may be annotated to obtain a label of the sample image. During the annotation process, one or more sample objects may be determined on the sample image. The category information of each sample object may be determined. The sample detection frame of each sample object may be determined. The sample detection frame may mark the area where each sample object is located in the sample image. In one example, the sample detection frame includes the coordinates of the four vertices of the detection frame. In another example, the sample detection frame includes the coordinates of the center point of the detection frame, the width of the detection frame, and the height of the detection frame.

[0032] For another example, the sample object may include a virtual image, a part of the virtual image (such as a hand, a head, etc.), etc. In one example, the virtual image may correspond to an object such as a person, an animal, a plant, a machine, etc. There may be at least one virtual image.

[0033] For example, a target detection model may be trained using a plurality of sample images to obtain a trained target detection model.

[0034] Figure 2 is a flowchart of a mutual information testing method according to an embodiment of the present disclosure.

[0035] like Figure 2 As shown, the method 200 may include operations S210 to S240.

[0036] In operation S210, target detection is performed on each of the multiple images to be tested from the interactive information to obtain multiple first detection results.

[0037] For example, interaction information may be generated when a user communicates and interacts with various smart devices.

[0038] In one example, a smart device may display an avatar.

[0039] In one example, taking the interactive process of querying the weather as an example, the user can send an audio corresponding to the voice message "Xiao Mou, how is the weather today?" The smart device in the test mode uses the audio corresponding to the voice message as input audio. In response to the input audio, the smart device can generate feedback information, which can be, for example, "It will rain today, and the maximum temperature is 30 degrees Celsius." Next, the smart device in the test mode can use the virtual image on the display screen to display the body movements and / or facial expressions corresponding to the feedback information.

[0040] For example, in order to test a smart device or program product with interactive functions, during the interactive process of querying the weather described above, within a certain period of time, all the content on the display screen of the smart device in the test mode can be recorded as a test video.

[0041] For example, the image to be tested may be input into the target detection model described above to obtain a first detection result.

[0042] For example, the first detection result includes a first detection frame of at least one object to be tested in the image to be tested.

[0043] For another example, the first detection frame may mark the area where each object to be tested is located in the image to be tested.

[0044] In operation S220, at least one first partial image is determined according to the first detection result.

[0045] For example, a first partial image is determined from the image to be tested according to the area marked by the first detection frame in the image to be tested.

[0046] In operation S230, an image comparison result is determined based on the at least one first partial image and the at least one second partial image.

[0047] For example, at least one second partial image is determined according to a second detection result, and the second detection result is obtained by performing target detection on a preset image. For example, the preset image can be input into the target detection model described above to obtain the second detection result.

[0048] For example, the preset image may be from a preset video.

[0049] In an example, the preset video may be recorded before obtaining the video to be tested, or may be recorded after obtaining the video to be tested, which is not limited in the present disclosure.

[0050] In one example, taking the interactive process of querying the weather as an example, in response to receiving the audio corresponding to the voice message "Xiao Mou, how is the weather today", the smart device in the preset mode uses the audio corresponding to the voice message as the input audio. In response to the input audio, the smart device can generate preset feedback information, which can be, for example, "It will rain today, and the maximum temperature is 30 degrees Celsius". Next, the smart device can use the virtual image on the display screen to display the preset body movements and / or preset facial expressions corresponding to the preset feedback information.

[0051] In one example, in the interactive process of querying the weather described above, within a certain period of time, for a smart device in a preset mode, all the content on its display screen can be recorded as a preset video.

[0052] For another example, the second detection frame may mark the area where each preset object is located in the preset image.

[0053] For example, a second partial image is determined from the preset image according to the area marked by the second detection frame in the preset image.

[0054] For example, for an image to be tested, at least one local similarity between a first partial image and at least one second partial image may be determined. The largest local similarity among the at least one similarity is taken as the local similarity corresponding to the first partial image. For at least one first partial image, when the local similarities corresponding to the first partial image are all greater than a preset local similarity threshold, the image to be tested may be taken as a valid image.

[0055] For example, the number of valid images may be used as the dividend, and the total number of images to be tested may be used as the divisor to perform a division operation to obtain an image comparison result.

[0056] In operation S240, a test result is determined according to the image comparison result.

[0057] For example, in a case where the image comparison result is greater than a preset image comparison threshold, the determined test result may indicate that the test passed.

[0058] Through the embodiments of the present disclosure, target detection is performed on the image to be tested from the interactive information, and comparison is performed based on the detection results. When a smart device displays multiple objects or multiple actions, automated testing can be implemented to reduce labor costs.

[0059] In some embodiments, the preset video and the video to be tested may be stored in a database.

[0060] In some embodiments, the object to be tested may include, for example, a virtual image, a part of the virtual image (such as a hand, a head, etc.), etc. In one example, the virtual image may correspond to an object such as a person, an animal, a plant, a machine, etc. There may be at least one virtual image.

[0061] For example, as described above, the first detection frame can mark the area where each object to be tested is located in the image to be tested. In one example, the first detection frame includes the coordinates of the four vertices of the first detection frame. In another example, the first detection frame includes the coordinates of the center point of the first detection frame, the width of the first detection frame, and the height of the first detection frame.

[0062] In some embodiments, the preset object may include, for example, a virtual image, a part of the virtual image (such as a hand, a head, etc.), etc. In one example, the virtual image may correspond to an object such as a person, an animal, a plant, a machine, etc. There may be at least one virtual image.

[0063] For example, as described above, the second detection frame can mark the area where each preset object is located in the preset image. In one example, the second detection frame includes the coordinates of the four vertices of the second detection frame. In another example, the second detection frame includes the coordinates of the center point of the second detection frame, the width of the second detection frame, and the height of the second detection frame.

[0064] In some embodiments, the plurality of images to be tested come from a video to be tested, and the video to be tested is determined according to the interaction information.

[0065] In some embodiments, performing target detection on multiple images to be tested from the interactive information to obtain multiple first detection results includes: extracting frames from the video to be tested to obtain multiple images to be tested; and performing target detection on the multiple images to be tested to obtain multiple first detection results.

[0066] For example, frame extraction can be performed according to the time information of the video to be tested. In one example, the duration of the video to be tested is 15 seconds, and a certain number of video frames can be extracted from the video corresponding to each second to obtain multiple images to be tested.

[0067] In some embodiments, target detection is performed on multiple images to be tested from the interactive information respectively, and obtaining multiple first detection results also includes: obtaining multiple second detection results, wherein the second detection result includes a second detection frame of at least one preset object in the preset image, and the multiple preset images are obtained by extracting frames from a preset video.

[0068] For example, frame extraction can be performed according to the time information of the preset video. In one example, the duration of the preset video is 15 seconds, and a certain number of video frames can be extracted from the video corresponding to each second to obtain a plurality of preset images.

[0069] In some embodiments, the trained target detection model described above can be used to perform target detection on the test image to obtain a first detection result. The trained target detection model described above can be used to perform target detection on the preset image to obtain a second detection result.

[0070] For example, the first detection result may further include category information of at least one object to be tested. For another example, the second detection result may further include category information of at least one preset object.

[0071] For another example, the at least one object to be tested may include one category of objects to be tested. Among the one category of objects to be tested, the maximum number of objects to be tested in the i-th category may be Ki.

[0072] In one example, after the image to be tested is input into the trained target detection model, the target detection model can determine at least one object to be tested in the image to be tested. There can be 1 category information of the object to be tested. In the first initial detection result output by the target detection model, the number of objects to be tested of the i-th category can be Hi. Hi is greater than or equal to Ki. In addition, in the first initial detection result, each object to be tested of the i-th category corresponds to a category probability value. Ki i-th category objects to be tested with larger category probability values ​​are retained, and other i-th category objects to be tested with smaller category probability values ​​are deleted from the first initial detection result to obtain the first detection result. I is an integer greater than or equal to 1. i is an integer greater than 1 or equal to 1, and i is an integer less than or equal to I. Hi is an integer greater than or equal to 1, and Ki is an integer greater than or equal to 1.

[0073] For another example, the at least one preset object may also include one category of preset objects. Among the one category of preset objects, the maximum number of preset objects of the i-th category may also be Ki.

[0074] In one example, after a preset image is input into a trained target detection model, the target detection model can determine at least one preset object in the preset image. There can be 1 category information of the preset object. In the second initial detection result output by the target detection model, the number of preset objects of the i-th category can be Hi. Hi is greater than or equal to Ki. In addition, in the second initial detection result, each preset object of the i-th category corresponds to a category probability value. Ki i-th category preset objects with larger category probability values ​​are retained, and other i-th category preset objects with smaller category probability values ​​are deleted from the second initial detection result to obtain a second detection result.

[0075] In some embodiments, the image to be tested matches a preset image, and the preset image matching the image to be tested is determined according to a plurality of first similarities between the image to be tested and a plurality of preset images.

[0076] In the embodiment of the present disclosure, a preset image matching the image to be tested may be determined according to a plurality of first similarities between the image to be tested and a plurality of preset images.

[0077] For example, the first similarity may be SSIM (Structural SIMilarity). For another example, the preset image having the greatest first similarity to the image to be tested may be used as the preset image that matches the image to be tested.

[0078] For another example, based on a sliding window strategy, the SSIM between the image to be tested and the preset image can be determined as the first similarity. In one example, a sliding window can be used to traverse each image to be tested and the preset image to calculate the SSIM of the two.

[0079] For another example, as described above, in the interactive process of querying the weather, all the contents on the display screen on the smart device in the test mode can be recorded as the video to be tested. In one example, all the contents on the display screen on the smart device in the test mode can be recorded as the initial video to be tested. For multiple images to be tested from the initial video to be tested, after determining multiple first similarities between the image to be tested Img_t' and multiple preset images, it can be determined whether the largest first similarity S1_max among the multiple first similarities is greater than the preset initial similarity threshold. In the case where the first similarity S1_max is less than the preset initial similarity threshold, the image to be tested Img_t' can be deleted from the multiple images to be tested from the initial video to be tested. Each image to be tested from the initial video to be tested is processed in a similar manner, and the video to be tested can be obtained based on the initial video to be tested.

[0080] Through the embodiments of the present disclosure, after obtaining a preset image matching each image to be tested, the video to be tested can be aligned with the preset video.

[0081] In some embodiments, based on the first detection result, determining at least one first partial image includes: for the object to be tested, determining a first target preset object from a preset image that matches the image to be tested, wherein category information of the first target preset object is consistent with category information of the object to be tested, and the first target preset object comes from at least one preset object; based on a first detection frame of the object to be tested, determining a first partial image; and based on a second detection frame of the first target preset object, determining a second partial image.

[0082] For example, in the image to be tested Img_t, the category information of the object to be tested Obj_t1 may indicate that the object to be tested Obj_t1 is the left hand of the virtual image; and the category information of the object to be tested Obj_t2 may indicate that the object to be tested Obj_t2 is the head of the virtual image.

[0083] For another example, in the preset image Img_ref1 matching the image to be tested Img_t, the category information of the preset object Obj_r11 may indicate that the preset object Obj_r11 is the left hand of the avatar; the category information of the preset object Obj_r12 may indicate that the preset object Obj_r12 is the head of the avatar.

[0084] For another example, as described above, the category information of the object to be tested Obj_t1 is consistent with the category information of the preset object Obj_r11. For the object to be tested Obj_t1, the preset object Obj_r11 can be used as the first target preset object. Next, according to the first detection frame of the object to be tested Obj_t1, the first partial image Img_t_P1 can be determined. According to the second detection frame of the preset object Obj_r11, the second partial image Img_ref1_P1 can be determined.

[0085] For another example, as described above, the category information of the object to be tested Obj_t2 is consistent with the category information of the preset object Obj_r12. For the object to be tested Obj_t2, the preset object Obj_r12 can be used as the first target preset object. Next, according to the first detection frame of the object to be tested Obj_t2, the first partial image Img_t_P2 can be determined. According to the second detection frame of the preset object Obj_r12, the second partial image Img_ref1_P2 can be determined.

[0086] It is understood that after determining at least one first partial image and at least one second partial image, an image comparison result may be determined. The image comparison result may include, for example, a first sub-comparison result. Figure 3 One way to determine the results of an image comparison is described in detail.

[0087] Figure 3 is a flowchart of determining an image comparison result according to another embodiment of the present disclosure.

[0088] like Figure 3 As shown, the method 330 can determine the image comparison result according to at least one first partial image and at least one second partial image. The following will be described in detail in conjunction with operations S331 to S333.

[0089] In operation S331, at least one second similarity is obtained based on at least one first partial image and at least one second partial image.

[0090] For example, the at least one first partial image may include the first partial image Img_t_P1 and the first partial image Img_t_P2 described above. For another example, the at least one second partial image may include the second partial image Img_ref1_P1 and the second partial image Img_ref1_P2 described above.

[0091] For another example, the second similarity S21 may be obtained based on the first partial image Img_t_P1 and the second partial image Img_ref1_P1. For another example, the second similarity S22 may be obtained based on the first partial image Img_t_P2 and the second partial image Img_ref1_P2.

[0092] In operation S332, in response to determining that the second similarity is less than the first preset similarity threshold, the image to be tested is determined as a first target image to be tested.

[0093] For example, the first preset similarity threshold may be 0.95.

[0094] For example, in response to at least one of the second similarity S21 and the second similarity S22 being less than a first preset similarity threshold, the image to be tested Img_t described above is used as a first target image to be tested.

[0095] Next, operations S331 and S332 may be performed on a plurality of images to be tested, so as to determine whether each image to be tested can be used as a first target image to be tested.

[0096] In operation S333, a first sub-comparison result is determined according to the number of the images to be tested and the number of the first target images to be tested.

[0097] For example, the total number of images to be tested is 100, and the number of first target images to be tested is 2. The difference between the two is the number of images to be tested whose similarity is greater than or equal to the first preset similarity threshold. In one example, the difference (e.g., 98) is used as the dividend, and the total number of images to be tested (e.g., 100) is used as the divisor to perform a division operation, and a first sub-comparison result (e.g., 0.98) can be obtained.

[0098] Through the embodiments of the present disclosure, between the image to be tested and the preset image matching the image to be tested, it can be determined whether the preset object is consistent with the object to be tested, so that the test can be performed more accurately, resulting in more accurate test results.

[0099] For example, as described above, the preset image Img_ref1 matches the image to be tested Img_t, and the overall similarity between the two is high. Determine whether the object to be tested Obj_t1 and the preset object Obj_r11 are consistent, and determine whether the preset image Img_ref1 is consistent with the local area related to the object in the image to be tested Img_t. If the object to be tested Obj_t1 and the preset object Obj_r11 are inconsistent, it can be considered that there is an error between the image to be tested Img_t and the preset image Img_ref1. The test can be performed more accurately, making the test result more accurate.

[0100] In some embodiments, a test result may be determined based on the first sub-comparison result.

[0101] For example, the test result may be determined based on the first sub-comparison result and the first preset image comparison threshold. In one example, when the first sub-comparison result is greater than or equal to the first preset image comparison threshold, the determined test result may indicate that the test passed. In one example, the first preset image comparison threshold may be a value such as 0.95 or 1.

[0102] In some embodiments, based on the first detection result, determining at least one first partial image includes: determining at least one target preset image from multiple preset images other than the preset image matching the image to be tested based on time information of the preset image matching the image to be tested; determining a second target preset object from the target preset image for the object to be tested, wherein category information of the second target preset object is consistent with category information of the object to be tested, and the second target preset object comes from at least one preset object in the target preset image; determining the first partial image based on the first detection frame of the object to be tested; and determining the second partial image based on the second detection frame of the second target preset object.

[0103] For example, as described above, the image to be tested Img_t matches Img_ref1. The time information of the image to be tested Img_t may indicate that the image to be tested Img_t is the second video frame extracted from the first second of the video to be tested.

[0104] For another example, the time information of the preset image Img_ref0 may indicate that the preset image Img_ref0 is the first video frame extracted from the first second of the preset video. The preset image Img_ref0 may be used as the target preset image.

[0105] For example, as described above, in the tested image Img_t, the category information of the tested object Obj_t1 may indicate that the tested object Obj_t1 is the left hand of the virtual image; the category information of the tested object Obj_t2 may indicate that the tested object Obj_t2 is the head of the virtual image.

[0106] For another example, in the preset image Img_ref0, the category information of the preset object Obj_r01 may indicate that the preset object Obj_r01 is the left hand of the avatar; and the category information of the preset object Obj_r02 may indicate that the preset object Obj_r02 is the head of the avatar.

[0107] For another example, as described above, the category information of the object to be tested Obj_t1 is consistent with the category information of the preset object Obj_r01. For the object to be tested Obj_t1, the preset object Obj_r01 can be used as the second target preset object. Next, according to the first detection frame of the object to be tested Obj_t1, the first partial image Img_t_P1 can be determined. According to the second detection frame of the preset object Obj_r01, the second partial image Img_ref0_P1 can be determined.

[0108] For another example, as described above, the category information of the object to be tested Obj_t2 is consistent with the category information of the preset object Obj_r02. For the object to be tested Obj_t2, the preset object Obj_r02 can be used as the second target preset object. Next, according to the first detection frame of the object to be tested Obj_t2, the first partial image Img_t_P2 can be determined. According to the second detection frame of the preset object Obj_r02, the second partial image Img_ref0_P2 can be determined.

[0109] It is understood that after determining at least one first partial image and at least one second partial image, an image comparison result may be determined. The image comparison result may include, for example, a second sub-comparison result. Figure 4 One way to determine the results of an image comparison is described in detail.

[0110] Figure 4 is a flowchart of determining an image comparison result according to another embodiment of the present disclosure.

[0111] like Figure 4 As shown, the method 430 can determine the image comparison result according to at least one first partial image and at least one second partial image. The following will be described in detail in conjunction with operations S434 to S436.

[0112] In operation S434, at least one third similarity is obtained based on the at least one first partial image and the at least one second partial image.

[0113] For example, the at least one first partial image may include the first partial image Img_t_P1 and the first partial image Img_t_P2 described above. For another example, the at least one second partial image may include the second partial image Img_ref0_P1 and the second partial image Img_ref1_P2 described above.

[0114] For another example, the third similarity S3_1 may be obtained according to the first partial image Img_t_P1 and the second partial image Img_ref0_P1. For another example, the third similarity S3_2 may be obtained according to the first partial image Img_t_P2 and the second partial image Img_ref0_P2.

[0115] In operation S435 , in response to determining that at least one third similarity is greater than or equal to a second preset similarity threshold, the image to be tested is determined as a second target image to be tested.

[0116] For example, the second preset similarity threshold may be 0.95.

[0117] For example, in response to the third similarity S3_1 and the third similarity S3_2 being both greater than or equal to the second preset similarity threshold, the image to be tested Img_t described above is used as a second target image to be tested.

[0118] Next, operations S434 and S435 may be performed on a plurality of images to be tested, so as to determine whether each image to be tested can be used as a second target image to be tested.

[0119] In operation S436 , a second sub-comparison result is determined according to the number of the images to be tested and the number of the second target images to be tested.

[0120] For example, the total number of images to be tested is 100, and the number of second target images to be tested is 10. The difference between the two is the number of images to be tested whose similarity is less than the second preset similarity threshold. In one example, the difference (e.g., 90) is used as the dividend, and the total number of images to be tested (e.g., 100) is used as the divisor to perform a division operation, and a second sub-comparison result (e.g., 0.90) can be obtained.

[0121] In some embodiments, a test result may be determined based on the second sub-comparison result.

[0122] For example, the test result may be determined based on the second sub-comparison result and the second preset image comparison threshold. In one example, when the second sub-comparison result is greater than or equal to the second preset image comparison threshold, the determined test result may indicate that the test passed. In one example, the second preset image comparison threshold may be a value such as 0.95 or 1.

[0123] Through the embodiments of the present disclosure, it is possible to determine whether a video to be tested has a freeze, and the test can be performed more accurately, making the test result more accurate.

[0124] For example, as described above, the preset image Img_ref1 matches the image to be tested Img_t. For the preset image Img_ref0, the preset image Img_ref1, and the image to be tested Img_t, the similarity between the preset image Img_ref1 and the image to be tested Img_t is high, while the similarity between the preset image Img_ref0 and the image to be tested Img_t is low, and the object in the preset image Img_ref1 and the object in the image to be tested Img_t may be consistent. The preset image Img_ref0 may be the first video frame extracted from the video of the first second in the preset video. The preset image Img_ref1 may be the second video frame extracted from the video of the first second in the preset video. The image to be tested Img_t may be the second video frame extracted from the video of the first second in the video to be tested. For a smooth video, the object in the first video frame and the object in the second video frame may be inconsistent.

[0125] Determine whether the object to be tested Obj_t1 is consistent with the preset object Obj_r01, and determine whether the object to be tested Obj_t2 is consistent with the preset object Obj_r02, and determine whether the preset image Img_ref0 is consistent with the local area related to the object in the image to be tested Img_t. If the preset image Img_ref0 is consistent with the local area related to the object in the image to be tested Img_t, it can be determined that the object in the preset image Img_ref0 and the object in the image to be tested Img_t have not changed, and the video to be tested has been stuck.

[0126] As described above, the test result can be determined based on the image data. It is understood that the test result can also be determined based on the audio data. This will be described in detail below.

[0127] In some embodiments, taking the interactive process of querying the weather as an example, the user can send a voice message "Xiao Mou, how is the weather today?" In response to receiving the voice message, the smart device in the test mode uses the audio corresponding to the voice message as the input audio. In response to the input audio, the smart device can generate feedback information, which can be, for example, "It will rain today, and the maximum temperature is 30 degrees Celsius." Next, the smart device in the test mode can use the virtual image on the display screen to display the body movements and / or facial expressions corresponding to the feedback information. In addition, the feedback audio corresponding to the feedback information can also be played.

[0128] For example, as described above, during the interactive process of querying the weather, within a certain period of time, for a smart device in a test mode, all the content on its display screen can be recorded and related audio data can be collected as a test video.

[0129] In some embodiments, taking the interactive process of querying the weather as an example, in response to receiving the voice message "Xiao Mou, how is the weather today", the smart device in the preset mode uses the audio corresponding to the voice message as the input audio. In response to the input audio, the smart device can generate preset feedback information, which can be, for example, "It will rain today, and the maximum temperature is 30 degrees Celsius". Next, the smart device can use the virtual image on the display screen to display the preset body movements and / or preset facial expressions corresponding to the preset feedback information. In addition, the preset feedback audio corresponding to the feedback information can also be played.

[0130] In one example, as described above, during the interactive process of querying the weather, within a certain period of time, for a smart device in a preset mode, all the content on its display screen can be recorded and related audio data can be collected as a preset video.

[0131] The following will be combined Figure 5 Determining test results using collected relevant audio data is described in detail.

[0132] Figure 5 is a flow chart of determining a test result according to another embodiment of the present disclosure.

[0133] like Figure 5 As shown, method 540 can determine the test result according to the image comparison result. The following will be described in detail in conjunction with operations S541 to S545.

[0134] In operation S541, at least one audio to be tested is obtained according to the video to be tested.

[0135] For example, the VAD (Voice Activity Detection) technology may be used to process the audio data of the video to be tested to obtain at least one audio to be tested.

[0136] In operation S542, at least one preset audio is obtained according to the preset video.

[0137] For example, the audio to be tested corresponds to a preset audio.

[0138] For example, the audio data of a preset video may be processed using VAD technology to obtain at least one preset audio.

[0139] For example, taking the duration of the video to be tested as 15 seconds, the time information of the audio to be tested indicates that the audio to be tested Voc_t exists from the 6th second to the 15th second. The time information of the preset audio Voc_ref corresponding to the audio to be tested Voc_t may indicate that the preset audio Voc_ref exists from the 6th second to the 15th second. It can be understood that, as described above, after the video to be tested is aligned with the preset video, the preset audio Voc_ref corresponding to the audio to be tested Voc_t may be obtained.

[0140] In operation S543, speech recognition is performed on at least one audio to be tested and at least one preset audio, respectively, to obtain at least one audio text to be tested and at least one preset audio text.

[0141] In the disclosed embodiment, the audio to be tested is divided to obtain input audio to be tested and feedback audio to be tested; speech recognition is performed on the input audio to be tested to obtain input text to be tested; and speech recognition is performed on the feedback audio to be tested to obtain feedback text to be tested.

[0142] For example, the Echo Cancellation algorithm can be used to divide the test audio Voc_t to obtain the test input audio Voc_t_in and the test feedback audio Voc_t_out. Then, various speech recognition models are used to perform speech recognition on the test input audio Voc_t_in and the test feedback audio Voc_t_out to obtain the test input text TVoc_t_in and the test feedback text TVoc_t_out.

[0143] In an embodiment of the present disclosure, the preset audio is divided to obtain preset input audio and preset feedback audio; speech recognition is performed on the preset input audio to obtain preset input text; and speech recognition is performed on the preset feedback audio to obtain preset feedback text.

[0144] For example, the preset audio Voc_ref can be divided by using the Echo Cancellation algorithm to obtain the preset input audio Voc_ref_in and the preset feedback audio Voc_ref_out. Then, various speech recognition models are used to perform speech recognition on the preset input audio Voc_ref_in and the preset feedback audio Voc_ref_out to obtain the preset input text TVoc_ref_in and the preset feedback text TVoc_ref_out.

[0145] In operation S544, an audio text comparison result is determined according to the at least one audio text to be tested and the at least one preset audio text.

[0146] In the embodiment of the present disclosure, the preset audio corresponding to the audio to be tested is used as the target preset audio.

[0147] For example, as described above, the audio to be tested Voc_t corresponds to the preset audio Voc_ref. The preset audio Voc_ref can be used as the target preset audio.

[0148] In the embodiment of the present disclosure, the input text to be tested of the input audio to be tested is matched with the preset input text of the target preset audio to obtain a first matching result.

[0149] For example, the number of identical characters in the input text to be tested TVoc_t_in and the preset input text TVoc_ref_in can be used as the dividend, and the number of characters in the preset input text TVoc_ref_in can be used as the divisor to perform a division operation to obtain the input text accuracy. If the input text accuracy is greater than the preset input text accuracy threshold, it can be determined that the first matching result indicates a successful match.

[0150] In the embodiment of the present disclosure, the feedback text to be tested of the input audio to be tested is matched with the preset feedback text of the target preset audio to obtain a second matching result.

[0151] For example, the number of identical characters in the feedback text to be tested TVoc_t_out and the preset feedback text TVoc_ref_out can be used as the dividend, and the number of characters in the preset feedback text TVoc_ref_out can be used as the divisor to perform a division operation to obtain the feedback text accuracy. If the feedback text accuracy is greater than the preset feedback text accuracy threshold, it can be determined that the second matching result indicates a successful match.

[0152] In an embodiment of the present disclosure, in response to determining that at least one of the first matching result and the second matching result indicates a matching failure, the audio to be tested is determined as the target audio to be tested; and the audio-text comparison result is determined based on the number of target audio to be tested and the number of audio to be tested.

[0153] For example, the number of target audios to be tested can be 0, and the number of audios to be tested can be 1. The difference between the two is used as the dividend, and the number of audios to be tested is used as the divisor to perform a division operation to obtain the audio text comparison result.

[0154] In operation S545, a test result is determined according to the audio-text comparison result and the image comparison result.

[0155] For example, the test result may be determined based on the audio-text comparison result and a preset audio-text comparison threshold.

[0156] In one example, as described above, the image comparison result includes a first comparison result and a second comparison result. When the first sub-comparison result is greater than or equal to a first preset image comparison threshold, the second sub-comparison result is greater than or equal to a second preset image comparison threshold, and the audio text comparison result is greater than or equal to a preset audio text comparison threshold, the determined test result may indicate that the test passed.

[0157] In some embodiments, the audio to be tested or the preset audio may be subjected to noise reduction processing to delete noise irrelevant to the input audio or the feedback audio.

[0158] As described above, the test result can be determined based on the image data and the audio data. It is understood that the test result can also be determined based on the text data. This will be described in detail below.

[0159] In some embodiments, taking the interactive process of querying the weather as an example, the user can send a voice message "Xiao Mou, how is the weather today?" In response to receiving the voice message, the smart device in the test mode uses the audio corresponding to the voice message as the input audio. In response to the input audio, the smart device can generate feedback information, which can be, for example, "It will rain today, and the maximum temperature is 30 degrees Celsius." Next, the smart device in the test mode can use the virtual image on the display screen to display the body movements and / or facial expressions corresponding to the feedback information, and can play the feedback audio corresponding to the feedback information. In addition, the text data corresponding to the feedback information can also be displayed on the display screen.

[0160] For example, as described above, during the interactive process of querying the weather, within a certain period of time, for a smart device in a test mode, all the content on its display screen can be recorded and related audio data can be collected as a test video.

[0161] In some embodiments, taking the interactive process of querying the weather as an example, in response to receiving the voice message "Xiao Mou, how is the weather today", the smart device in the preset mode uses the audio corresponding to the voice message as the input audio. In response to the input audio, the smart device can generate preset feedback information, and the preset feedback information can be, for example, "It will rain today, and the maximum temperature is 30 degrees Celsius". Next, the smart device can use the virtual image on the display screen to display the preset body movements and / or preset facial expressions corresponding to the preset feedback information, and can play the preset feedback audio corresponding to the feedback information. In addition, the text data corresponding to the preset feedback information can also be displayed on the display screen.

[0162] In one example, as described above, during the interactive process of querying the weather, within a certain period of time, for a smart device in a preset mode, all the content on its display screen can be recorded and related audio data can be collected as a preset video.

[0163] The following will be combined Figure 6 Describe in detail the use of text data to determine test results.

[0164] Figure 6 is a flow chart of determining a test result according to another embodiment of the present disclosure.

[0165] like Figure 6 As shown, method 640 can determine the test result according to the image comparison result. The following will be described in detail in conjunction with operations S646 to S648.

[0166] In operation S646, text recognition is performed on the preset image matching the image to be tested and the image to be tested, respectively, to obtain the preset image text and the image to be tested text.

[0167] For example, the CTC (Connectionist Temporal Classification) technology may be used to perform text recognition on the image to be tested Img_t and the preset image Img_ref1 to obtain the preset image text TImg_ref1 and the image to be tested text TImg_t.

[0168] In operation S647, an image text comparison result is determined according to the preset image text and the image text to be tested.

[0169] For example, the number of identical characters between the image text to be tested TImg_t and the preset image text TImg_ref1 may be determined.

[0170] Next, operations S434 and S435 may be performed on the multiple images to be tested to determine the total number of identical characters in the multiple image texts to be tested and the multiple preset image texts. The total number of identical characters is then used as the dividend and the total number of characters in the multiple preset image texts is used as the divisor to perform a division operation to obtain an image text comparison result.

[0171] In operation S648, a test result is determined according to the image-text comparison result and the image comparison result.

[0172] For example, the test result may be determined based on the image-text comparison result and a preset image-text comparison threshold.

[0173] In one example, as described above, the image comparison result includes a first comparison result and a second comparison result. When the first sub-comparison result is greater than or equal to a first preset image comparison threshold, the second sub-comparison result is greater than or equal to a second preset image comparison threshold, and the image text comparison result is greater than or equal to the preset image text comparison threshold, the determined test result may indicate that the test passed.

[0174] In some embodiments, the test result may be determined based on the audio text comparison result, the image text comparison result, and the image comparison result.

[0175] For example, as described above, the image comparison result includes a first comparison result and a second comparison result. When the first sub-comparison result is greater than or equal to a first preset image comparison threshold, the second sub-comparison result is greater than or equal to a second preset image comparison threshold, the audio text comparison result is greater than a preset audio text comparison threshold, and the image text comparison result is greater than or equal to a preset image text comparison threshold, the determined test result may indicate that the test passed.

[0176] In some embodiments, determining the test result according to the image comparison result includes: determining the test result according to the image comparison result when the duration information of the video to be tested is consistent with the duration information of the preset video.

[0177] For example, if the duration of the video to be tested is 15 seconds and the duration of the preset video is 15 seconds, it can be determined that the two are consistent. Next, the test result can be determined based on the image comparison result.

[0178] For another example, if the duration of the video to be tested is longer than the duration of the preset video, it can be determined that the test has failed and that the smart device has been stuck.

[0179] For another example, if the duration of the video to be tested is shorter than the duration of the preset video, it can be determined that the operation of the smart device is abnormal or some functions are not implemented.

[0180] In some embodiments, the time information of the video to be tested may include multiple time periods.

[0181] For example, in time period t1 (e.g., from the 1st second to the 5th second), the object to be tested Obj_1 appears in the video to be tested. There is no audio and text in the video to be tested. In time period t2 (e.g., from the 6th second to the 8th second), the object to be tested Obj_2 appears in the video to be tested. In addition, input audio appears in the video to be tested, and text corresponding to the input audio appears in the video to be tested. In time period t3 (e.g., from the 9th second to the 15th second), the object to be tested Obj_3 appears in the video to be tested. In addition, feedback audio appears in the video to be tested, and text corresponding to the feedback audio appears in the video to be tested.

[0182] In some embodiments, the time information of the preset video may include multiple time periods.

[0183] For example, in time period t1 (e.g., from the 1st second to the 5th second), a preset object Obj_1' appears in the preset video. There is no audio and text in the preset video. In time period t2 (e.g., from the 6th second to the 8th second), a preset object Obj_2' appears in the preset video. In addition, input audio appears in the preset video, and text corresponding to the input audio appears in the preset video. In time period t3 (e.g., from the 9th second to the 15th second), a preset object Obj_3' appears in the preset video. In addition, feedback audio appears in the preset video, and text corresponding to the feedback audio appears in the preset video.

[0184] Furthermore, in one example, during the time period t1 , the image comparison result of the time period t1 may be determined according to the image to be tested and the preset image using the methods 330 and 430 described above.

[0185] In time period t2, the image comparison result of time period t2 can be determined according to the image to be tested and the preset image using methods 330 and 430 described above. The audio text comparison result of time period t2 and the image text comparison result of time period t2 can also be determined according to methods 540 and 640 described above.

[0186] In time period t3, the image comparison result of time period t3 can be determined according to the image to be tested and the preset image using methods 330 and 430 described above. The audio text comparison result of time period t3 and the image text comparison result of time period t3 can also be determined according to methods 540 and 640 described above.

[0187] Next, when the image comparison result of each time period is greater than or equal to the preset image comparison threshold, the audio-text comparison result of each time period is greater than the preset audio-text comparison result, and the image-text comparison result of each time period is greater than or equal to the preset image-text comparison threshold, the determined test result can indicate that the test passed.

[0188] The method provided by the present disclosure is described in detail above. The method provided by the present disclosure will be further described in detail below in conjunction with the image to be tested and the first partial image.

[0189] Fig. 7A is a schematic diagram of an image to be tested according to an embodiment of the present disclosure.

[0190] like Fig. 7A As shown, the image to be tested 700 includes a virtual image 710 .

[0191] By inputting the image to be tested 700 into the target detection model, it is possible to obtain the object to be tested 711, the object to be tested 712, the first detection frame 721 of the object to be tested 711, and the first detection frame 722 of the object to be tested 712. For example, the object to be tested 711 may be the head of the virtual image 710, and the object to be tested 712 may be the right hand of the virtual image.

[0192] like Fig. 7A As shown, the image to be tested 700 also includes a text area 730 . The CTC technology is used to perform text recognition on the image to be tested 700 , and the obtained text to be tested may include multiple English characters in the text area 730 .

[0193] Figure 7B and Figure 7C is a schematic diagram of a first partial image according to an embodiment of the present disclosure.

[0194] like Figure 7B As shown, the first partial image 741 may be obtained according to the first detection frame 721. The first partial image 741 may correspond to the object to be tested 711.

[0195] like Figure 7C As shown, the first partial image 742 may be obtained according to the first detection frame 722. The first partial image 742 may correspond to the object to be tested 712.

[0196] Figure 8 is a block diagram of an interactive information testing device according to an embodiment of the present disclosure.

[0197] like Figure 8 As shown, the device 800 may include a target detection module 810 , a first determination module 820 , a second determination module 830 and a third determination module 840 .

[0198] The object detection module 810 is used to perform object detection on multiple images to be tested from the interactive information to obtain multiple first detection results. For example, the first detection result includes a first detection frame of at least one object to be tested in the image to be tested.

[0199] The first determination module 820 is configured to determine at least one first partial image according to the first detection result.

[0200] The second determination module 830 is used to determine the image comparison result according to at least one first partial image and at least one second partial image. For example, at least one second partial image is determined according to a second detection result, and the second detection result is obtained by performing target detection on a preset image.

[0201] The third determination module 840 is used to determine the test result according to the image comparison result.

[0202] In some embodiments, the multiple images to be tested are from the video to be tested, and the video to be tested is determined according to the interactive information. The target detection module includes: an acquisition submodule, which is used to obtain multiple second detection results, wherein the second detection result includes a second detection frame of at least one preset object in the preset image, and the multiple preset images are obtained by extracting frames from the preset video. A frame extraction submodule is used to extract frames from the video to be tested to obtain multiple images to be tested; and a target detection submodule is used to perform target detection on the multiple images to be tested respectively to obtain multiple first detection results.

[0203] In some embodiments, the image to be tested matches a preset image, and the preset image matching the image to be tested is determined according to a plurality of first similarities between the image to be tested and a plurality of preset images.

[0204] In some embodiments, the first determination module includes: a first determination submodule, used to determine, for the object to be tested, a first target preset object from a preset image that matches the image to be tested, wherein the category information of the first target preset object is consistent with the category information of the object to be tested, and the first target preset object comes from at least one preset object; a second determination submodule, used to determine a first partial image based on a first detection frame of the object to be tested; and a third determination submodule, used to determine a second partial image based on a second detection frame of the first target preset object.

[0205] In some embodiments, the image comparison result includes a first sub-comparison result, and the second determination module includes: a first acquisition sub-module, used to obtain at least one second similarity based on at least one first partial image and at least one second partial image; a fourth determination sub-module, used to determine the image to be tested as a first target image to be tested in response to determining that the second similarity is less than a first preset similarity threshold; and a fifth determination sub-module, used to determine the first sub-comparison result based on the number of images to be tested and the number of first target images to be tested.

[0206] In some embodiments, the first determination module includes: a sixth determination submodule, used to determine at least one target preset image from multiple preset images other than the preset image matching the image to be tested based on time information of the preset image matching the image to be tested; a seventh determination submodule, used to determine a second target preset object from the target preset image for the object to be tested, wherein the category information of the second target preset object is consistent with the category information of the object to be tested, and the second target preset object comes from at least one preset object in the target preset image; an eighth determination submodule, used to determine a first partial image based on a first detection frame of the object to be tested; and a ninth determination submodule, used to determine a second partial image based on a second detection frame of the second target preset object.

[0207] In some embodiments, the image comparison result includes a second sub-comparison result, and the second determination module includes: a second acquisition sub-module, used to obtain at least one third similarity based on at least one first partial image and at least one second partial image; a tenth determination sub-module, used to determine the image to be tested as a second target image to be tested in response to determining that at least one third similarity is greater than or equal to a second preset similarity threshold; and an eleventh determination sub-module, used to determine the second sub-comparison result based on the number of images to be tested and the number of second target images to be tested.

[0208] In some embodiments, the third determination module includes: a third acquisition submodule, used to obtain at least one audio to be tested based on the video to be tested; a fourth acquisition submodule, used to obtain at least one preset audio based on the preset video, wherein the audio to be tested corresponds to a preset audio; a speech recognition submodule, used to perform speech recognition on at least one audio to be tested and at least one preset audio, respectively, to obtain at least one audio text to be tested and at least one preset audio text; a twelfth determination submodule, used to determine the audio-text comparison result based on at least one audio text to be tested and at least one preset audio text; and a thirteenth determination submodule, used to determine the test result based on the audio-text comparison result and the image comparison result.

[0209] In some embodiments, the audio text to be tested includes an input text to be tested and a feedback text to be tested, the preset audio text includes a preset input text and a preset feedback text, and the speech recognition submodule includes: a first division unit, used to divide the audio to be tested to obtain the input audio to be tested and the feedback audio to be tested; a first speech recognition unit, used to perform speech recognition on the input audio to be tested to obtain the input text to be tested; a second speech recognition unit, used to perform speech recognition on the feedback audio to be tested to obtain the feedback text to be tested; a second division unit, used to divide the preset audio to obtain the preset input audio and the preset feedback audio; a third speech recognition unit, used to perform speech recognition on the preset input audio to obtain the preset input text; and a fourth speech recognition unit, used to perform speech recognition on the preset feedback audio to obtain the preset feedback text.

[0210] In some embodiments, the twelfth determination submodule includes: a first determination unit, used to take the preset audio corresponding to the audio to be tested as the target preset audio; a first matching unit, used to match the input text to be tested of the input audio to be tested with the preset input text of the target preset audio to obtain a first matching result; a second matching unit, used to match the feedback text to be tested of the input audio to be tested with the preset feedback text of the target preset audio to obtain a second matching result; a second determination unit, used to determine the audio to be tested as the target audio to be tested in response to determining that at least one of the first matching result and the second matching result indicates a match failure; and a third determination unit, used to determine the audio text comparison result based on the number of target audio to be tested and the number of audio to be tested.

[0211] In some embodiments, the third determination module includes: a text recognition submodule, used to perform text recognition on a preset image that matches the image to be tested and the image to be tested, respectively, to obtain the preset image text and the image text to be tested; a fourteenth determination submodule, used to determine the image-text comparison result based on the preset image text and the image text to be tested; and a fifteenth determination submodule, used to determine the test result based on the image-text comparison result and the image comparison result.

[0212] In some embodiments, the third determination unit is further configured to determine the test result according to the image comparison result when the duration information of the video to be tested is consistent with the duration information of the preset video.

[0213] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0214] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0215] Fig. 9 A schematic block diagram of an example electronic device 900 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0216] like Fig. 9As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0217] A number of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0218] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as the interactive information test method. For example, in some embodiments, the interactive information test method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the interactive information test method described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to perform the interactive information test method in any other appropriate manner (e.g., by means of firmware).

[0219] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0220] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0221] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0222] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0223] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0224] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0225] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0226] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for testing interactive information, comprising: Performing target detection on the multiple images to be tested from the interactive information respectively to obtain multiple first detection results, wherein the first detection result includes a first detection frame of at least one object to be tested in the image to be tested, and the category information of the object to be tested is used to indicate that the object to be tested is a virtual image or a part of the virtual image; Determine at least one first partial image according to the first detection result; determining an image comparison result according to the at least one first partial image and the at least one second partial image, wherein the at least one second partial image is determined according to a second detection result, the second detection result is obtained by performing target detection on a preset image, the second detection result includes a second detection frame of at least one preset object in the preset image, and the category information of the preset object is used to indicate that the preset object is the virtual image or a part of the virtual image; and Determine the test result according to the image comparison result, Wherein, determining at least one first partial image according to the first detection result includes: For the object to be tested, determining a first target preset object from a preset image matching the image to be tested, wherein category information of the first target preset object is consistent with category information of the object to be tested, and the first target preset object comes from the at least one preset object; Determining the first partial image in the image to be tested according to a first detection frame of the object to be tested; The second partial image is determined in the preset image matching the image to be tested according to a second detection frame of the first target preset object in a second detection result of the preset image matching the image to be tested.

2. The method according to claim 1, wherein: The multiple images to be tested are from a video to be tested, and the video to be tested is determined according to the interaction information. The performing target detection on the multiple images to be tested from the interactive information to obtain multiple first detection results includes: Acquire a plurality of the second detection results, wherein the plurality of preset images are obtained by extracting frames from a preset video; Extracting frames from the video to be tested to obtain the multiple images to be tested; and Object detection is performed on the multiple images to be tested respectively to obtain multiple first detection results.

3. The method according to claim 2, wherein: The image to be tested matches one of the preset images, and the preset image that matches the image to be tested is determined according to a plurality of first similarities between the image to be tested and a plurality of the preset images.

4. The method according to claim 3, wherein: The image comparison result includes a first sub-comparison result, The determining the image comparison result according to the at least one first partial image and the at least one second partial image comprises: Obtaining at least one second similarity according to the at least one first partial image and the at least one second partial image determined in a preset image matching the image to be tested; In response to determining that the second similarity is less than a first preset similarity threshold, determining the image to be tested as a first target image to be tested; and The first sub-comparison result is determined according to the number of the images to be tested and the number of the first target images to be tested.

5. The method according to claim 3, wherein: The determining, according to the first detection result, at least one first partial image further comprises: Determining at least one target preset image from a plurality of preset images other than the preset image matching the image to be tested according to time information of the preset image matching the image to be tested; For the object to be tested, determining a second target preset object from the target preset image, wherein category information of the second target preset object is consistent with category information of the object to be tested, and the second target preset object comes from at least one preset object in the target preset image; determining the first partial image in the image to be tested according to a first detection frame of the object to be tested; and According to a second detection frame of the second target preset object, the second partial image is determined in the target preset image.

6. The method according to claim 5, wherein: The image comparison result includes a second sub-comparison result, The determining the image comparison result according to the at least one first partial image and the at least one second partial image comprises: Obtaining at least one third similarity according to the at least one first partial image and the at least one second partial image determined in the target preset image; In response to determining that the at least one third similarity is greater than or equal to a second preset similarity threshold, determining the image to be tested as a second target image to be tested; and The second sub-comparison result is determined according to the number of the images to be tested and the number of the second target images to be tested.

7. The method according to claim 2 or 3, wherein: Determining the test result according to the image comparison result includes: Obtaining at least one audio to be tested according to the video to be tested; According to the preset video, at least one preset audio is obtained, wherein the audio to be tested corresponds to one of the preset audios; Performing speech recognition on the at least one audio to be tested and the at least one preset audio respectively to obtain at least one audio text to be tested and at least one preset audio text; Determining an audio text comparison result according to the at least one audio text to be tested and the at least one preset audio text; and The test result is determined according to the audio-text comparison result and the image comparison result.

8. The method according to claim 7, wherein: The audio text to be tested includes an input text to be tested and a feedback text to be tested, and the preset audio text includes a preset input text and a preset feedback text. The performing speech recognition on the at least one audio to be tested and the at least one preset audio respectively to obtain at least one audio text to be tested and at least one preset audio text comprises: Dividing the audio to be tested to obtain input audio to be tested and feedback audio to be tested; Performing speech recognition on the input audio to be tested to obtain the input text to be tested; Performing speech recognition on the feedback audio to be tested to obtain the feedback text to be tested; Dividing the preset audio to obtain preset input audio and preset feedback audio; Performing speech recognition on the preset input audio to obtain the preset input text; and Perform speech recognition on the preset feedback audio to obtain the preset feedback text.

9. The method according to claim 8, wherein: The step of determining the audio text comparison result according to the at least one audio text to be tested and the at least one preset audio text comprises: Using the preset audio corresponding to the audio to be tested as the target preset audio; Matching the input text to be tested of the input audio to be tested with the preset input text of the target preset audio to obtain a first matching result; Matching the to-be-tested feedback text of the to-be-tested input audio with the preset feedback text of the target preset audio to obtain a second matching result; In response to determining that at least one of the first matching result and the second matching result indicates a match failure, determining the audio to be tested as a target audio to be tested; and The audio-text comparison result is determined according to the number of the target audios to be tested and the number of the audios to be tested.

10. The method according to claim 3, wherein: Determining the test result according to the image comparison result includes: Performing text recognition on a preset image matching the image to be tested and the image to be tested respectively to obtain a preset image text and a test image text; Determining an image text comparison result according to the preset image text and the image text to be tested; and The test result is determined according to the image-text comparison result and the image comparison result.

11. The method according to any one of claims 2 to 6 and 8 to 10, wherein: Determining the test result according to the image comparison result includes: When the duration information of the video to be tested is consistent with the duration information of the preset video, a test result is determined according to the image comparison result.

12. An interactive information testing device, comprising: an object detection module, configured to perform object detection on each of the plurality of images to be tested from the interactive information, and obtain a plurality of first detection results, wherein the first detection result includes a first detection frame of at least one object to be tested in the image to be tested, and the category information of the object to be tested is used to indicate that the object to be tested is a virtual image or a part of the virtual image; A first determining module, configured to determine at least one first partial image according to the first detection result; a second determination module, configured to determine an image comparison result according to the at least one first partial image and the at least one second partial image, wherein the at least one second partial image is determined according to a second detection result, the second detection result is obtained by performing target detection on a preset image, the second detection result includes a second detection frame of at least one preset object in the preset image, and the category information of the preset object is used to indicate that the preset object is the virtual image or a part of the virtual image; and A third determination module is used to determine a test result according to the image comparison result. Wherein, the first determination module includes: A first determination submodule is used to determine, for the object to be tested, a first target preset object from a preset image matching the image to be tested, wherein the category information of the first target preset object is consistent with the category information of the object to be tested, and the first target preset object comes from the at least one preset object; A second determining submodule is configured to determine the first partial image in the image to be tested according to a first detection frame of the object to be tested; and The third determination submodule is used to determine the second partial image in the preset image matching the image to be tested according to the second detection frame of the first target preset object in the second detection result of the preset image matching the image to be tested.

13. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 11.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 11.

15. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 11.

Images