Human-computer interaction operation interface verification method and device, electronic equipment and storage medium

Abstract

The invention discloses a human-computer interaction operation interface verification method and device, electronic equipment and a storage medium. The method comprises the following steps: determining a plurality of man-machine interaction terminals to be subjected to interface verification; for each man-machine interaction end, collecting a real-time operation image of an interaction operation interface of the man-machine interaction end; image verification is conducted on the real-time operation image through the interface verification model, an image verification result is obtained, the image verification result represents the operation category of man-machine interaction operation corresponding to the real-time operation image of the interaction operation interface, and the operation category comprises normal operation and / or abnormal operation; and performing interface verification feedback on the man-machine interaction end based on the image verification result. Based on the technical scheme of the invention, the effect of carrying out automatic real-time operation verification on the man-machine interaction operation interfaces of a plurality of man-machine interaction terminals at the same time can be realized, the waste of human resources is reduced, the timeliness of verification is ensured, and meanwhile, the accuracy of a verification result can be ensured.

Description

Technical Field

[0001] The present invention relates to the field of computer application technology, and in particular to a human-computer interaction operation interface verification method, device, electronic equipment and storage medium. Background Art

[0002] Banking systems are characterized by high complexity and high professionalism. Therefore, clear execution standards are usually set for the interactive operations of operators performing operational tasks on the system to regulate the execution of interactive operations and avoid production incidents caused by operational errors.

[0003] In related technologies, the method of "one person operating, two people reviewing, and a dedicated person checking" is usually used to perform real-time verification of operator interaction operations to ensure standard execution of interaction operations. However, the method of multiple people performing interaction operation verification has problems such as waste of human resources and low timeliness. Summary of the invention

[0004] The present invention provides a human-computer interactive operation interface verification method, device, electronic device and storage medium to solve the technical problems of waste of human resources and low timeliness in the current interactive operation verification method.

[0005] According to one aspect of the present invention, a human-computer interaction operation interface verification method is provided, the method comprising:

[0006] Determine multiple human-computer interaction terminals to be verified by the interface;

[0007] For each of the human-computer interaction terminals, a real-time operation image of the interactive operation interface of the human-computer interaction terminal is collected, wherein the real-time operation image includes at least one of operation target information, interactive operation information, and interface feedback information;

[0008] Performing image verification on the real-time operation image through an interface verification model to obtain an image verification result, wherein the image verification result represents an operation category of the human-computer interaction operation corresponding to the real-time operation image of the interactive operation interface, and the operation category includes normal operation and / or abnormal operation;

[0009] Based on the image verification result, interface verification feedback is performed on the human-computer interaction terminal.

[0010] According to another aspect of the present invention, a human-computer interaction operation interface verification device is provided, the device comprising:

[0011] An interaction terminal determination module, used to determine multiple human-computer interaction terminals to be interface verified;

[0012] An image acquisition module, used for acquiring a real-time operation image of the interactive operation interface of each human-computer interaction terminal, wherein the real-time operation image includes at least one of operation target information, interactive operation information and interface feedback information;

[0013] An image verification module, used to perform image verification on the real-time operation image through an interface verification model to obtain an image verification result, wherein the image verification result represents an operation category of the human-computer interaction operation corresponding to the real-time operation image of the interactive operation interface, and the operation category includes normal operation and / or abnormal operation;

[0014] A verification feedback module is used to provide interface verification feedback to the human-computer interaction terminal based on the image verification result.

[0015] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0016] at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the human-computer interaction operation interface verification method described in any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the human-computer interaction operation interface verification method described in any embodiment of the present invention when executed.

[0020] The technical solution of the embodiment of the present invention is to determine multiple human-computer interaction terminals to be interface verified; for each of the human-computer interaction terminals, collect the real-time operation image of the interactive operation interface of the human-computer interaction terminal, wherein the real-time operation image includes at least one of the operation target information, interactive operation information and interface feedback information; perform image verification on the real-time operation image through an interface verification model to obtain an image verification result, wherein the image verification result characterizes the operation category of the human-computer interaction operation corresponding to the real-time operation image of the interactive operation interface, and the operation category includes normal operation and / or abnormal operation; and perform interface verification feedback on the human-computer interaction terminal based on the image verification result. The present invention can achieve the effect of automatically performing real-time operation verification on the human-computer interaction operation interfaces of multiple human-computer interaction terminals at the same time. The present invention can reduce the waste of human resources and ensure the timeliness of verification, while ensuring the accuracy of the verification result.

[0021] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 is a flow chart of a human-computer interaction operation interface verification method provided according to Embodiment 1 of the present invention;

[0024] Figure 2 is a flow chart of a human-computer interaction operation interface verification method provided according to Embodiment 2 of the present invention;

[0025] Figure 3 is a flowchart of a human-computer interaction operation interface verification method provided according to Embodiment 3 of the present invention;

[0026] Figure 4 is a structural schematic diagram of a human-computer interaction operation interface verification device provided according to a fourth embodiment of the present invention;

[0027] Figure 5 It is a structural schematic diagram of an electronic device for implementing the human-computer interactive operation interface verification method of an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] Embodiment 1

[0031] Figure 1 A flowchart of a human-computer interaction interface verification method is provided for the first embodiment of the present invention. This embodiment is applicable to the case where interactive operation verification is realized through interface verification. The method can be executed by a human-computer interaction interface verification device. The human-computer interaction interface verification device can be implemented in the form of hardware and / or software. The human-computer interaction interface verification device can be configured in a computer. Figure 1 As shown, the method includes:

[0032] S110, determining multiple human-computer interaction terminals to be interface verified.

[0033] The human-computer interaction terminal may be understood as a terminal having a human-computer interaction function.

[0034] In the embodiment of the present invention, the interaction terminal categories of different human-computer interaction terminals may be different or the same. Exemplarily, the interaction terminal categories may include management interaction terminals and / or business interaction terminals, etc. The interaction operation interfaces of the human-computer interaction terminals of different interaction terminal categories may be different.

[0035] In the embodiment of the present invention, the human-computer interaction operation interface can be verified simultaneously for a plurality of human-computer interaction terminals (the types of the interaction terminals can be the same or different).

[0036] S120: For each of the human-computer interaction terminals, collect a real-time operation image of the interactive operation interface of the human-computer interaction terminal.

[0037] The interactive operation interface may be understood as the operation interface of the human-computer interaction terminal.

[0038] In the embodiment of the present invention, the interactive operation interfaces of different human-computer interaction terminals may be the same or different.

[0039] The real-time operation image may be a real-time interface diagram of the interactive operation interface. In the implementation of the present invention, the real-time operation image includes at least one of operation target information, interactive operation information and interface feedback information.

[0040] The operation target information can be understood as information of the operation target. The operation target can be a target of an interactive operation. Exemplarily, the operation target can be A data in the interface, or B plug-in in the interface, etc.

[0041] The interactive operation information can be understood as information of the interactive operation. The interactive operation can be an execution operation of the user. The interactive operation information can be reflected based on the interface cursor corresponding to the operation mouse. Exemplarily, the interactive operation can be operations such as adding, deleting, modifying and checking.

[0042] The interface feedback information may be feedback information of the operation interface. The interface feedback information may be information fed back by the target operating system of the human-computer interaction terminal based on the interactive operation acting on the operation target. Exemplarily, the interface feedback information may be operation success or operation failure, etc.

[0043] Optionally, the collecting of the real-time operation image of the interactive operation interface of the human-computer interaction terminal includes:

[0044] An image acquisition cycle corresponding to the human-computer interaction terminal is determined, and a real-time operation image of the interactive operation interface of the human-computer interaction terminal is acquired according to the image acquisition cycle.

[0045] The image acquisition period can be understood as the image acquisition period. In the embodiment of the present invention, the image acquisition period can be preset according to the scene requirements and is not specifically limited here. Exemplarily, the image acquisition period can be 0.1 seconds, 0.2 seconds or 0.5 seconds, etc.

[0046] Based on the above-mentioned embodiment scheme, the present invention realizes the real-time automatic acquisition of the real-time operation image, so as to perform the effect of real-time automatic verification of the human-computer interactive operation interface.

[0047] S130: Performing image verification on the real-time operation image through an interface verification model to obtain an image verification result.

[0048] The interface verification model may have the function of verifying the interface operation image. In an embodiment of the present invention, the interface verification model may be obtained by training a convolutional neural network using a training sample set.

[0049] The image verification result may represent an operation category of the human-computer interactive operation corresponding to the real-time operation image of the interactive operation interface, where the operation category includes a normal operation and / or an abnormal operation.

[0050] Optionally, performing image verification on the real-time operation image through an interface verification model to obtain an image verification result includes:

[0051] Determining an image category of the real-time operation image through an interface verification model, wherein the image category is related to an interaction terminal category of the human-computer interaction terminal corresponding to the real-time operation image;

[0052] Performing image verification on the real-time operation image based on the image category to obtain an image verification result.

[0053] In the embodiment of the present invention, the interface verification model performs image verification based on the interface verification model. That is, for the real-time operation images of different image categories, the rules for image verification by the interface verification model may be different.

[0054] In an embodiment of the present invention, the image category and the interactive terminal category can be mapped one to one. That is, for the real-time operation images of the human-computer interactive terminals of the same interactive terminal category, the verification rules of the interface verification model can be the same. For the real-time operation images of the human-computer interactive terminals of different interactive terminal categories, the verification rules of the interface verification model can be different.

[0055] Based on the above-mentioned embodiment scheme, the present invention can perform targeted image verification on the real-time operation image based on the image category, thereby improving the accuracy of image verification.

[0056] Optionally, the image verification result includes verification passed and / or verification failed; performing image verification on the real-time operation image based on the image category to obtain the image verification result includes:

[0057] Extracting text information from the real-time operation image to determine key text information;

[0058] In a case where the key text information satisfies a target text condition, the image verification result of the real-time operation image is determined to be the verification passed, wherein the target text condition is related to the image category of the real-time operation image.

[0059] The key text information may be text information in the real-time operation image obtained by information extraction. Optionally, the key text information may include at least one of operation target information, interactive operation information and interface feedback information.

[0060] Optionally, the target text condition may be that the key text information includes operation target information, interactive operation information and interface feedback information at the same time, and the value corresponding to each information is a target value preset in advance.

[0061] In the embodiment of the present invention, the target text conditions corresponding to different image categories may be different or the same. For example, for different image categories, the target value preset in advance for each information may be different.

[0062] Based on the above embodiment scheme, image verification is performed by extracting text information, which improves the accuracy of image verification.

[0063] S140: Performing interface verification feedback on the human-computer interaction terminal based on the image verification result.

[0064] In the embodiment of the present invention, the image verification result includes verification passed and / or verification failed.

[0065] Optionally, when the image verification result is the verification failure, the image verification result is fed back to the interactive operation interface of the human-computer interaction terminal.

[0066] The technical solution of the embodiment of the present invention is to determine multiple human-computer interaction terminals to be interface verified; for each of the human-computer interaction terminals, collect the real-time operation image of the interactive operation interface of the human-computer interaction terminal, wherein the real-time operation image includes at least one of the operation target information, interactive operation information and interface feedback information; perform image verification on the real-time operation image through an interface verification model to obtain an image verification result, wherein the image verification result characterizes the operation category of the human-computer interaction operation corresponding to the real-time operation image of the interactive operation interface, and the operation category includes normal operation and / or abnormal operation; and perform interface verification feedback on the human-computer interaction terminal based on the image verification result. The present invention can achieve the effect of automatically performing real-time operation verification on the human-computer interaction operation interfaces of multiple human-computer interaction terminals at the same time. The present invention can reduce the waste of human resources and ensure the timeliness of verification, while ensuring the accuracy of the verification result.

[0067] Embodiment 2

[0068] Figure 2 This is a flowchart of a human-computer interaction operation interface verification method provided in the second embodiment of the present invention. This embodiment is to supplement the image verification of the real-time operation image through the interface verification model described in the above embodiment. Figure 2 As shown, the method includes:

[0069] S210, determining multiple human-computer interaction terminals to be interface verified.

[0070] S220: For each of the human-computer interaction terminals, collect a real-time operation image of the interactive operation interface of the human-computer interaction terminal.

[0071] S230: Determine a historical operation image corresponding to the real-time operation image, and determine image similarity between the real-time operation image and the historical operation image.

[0072] The historical operation image may be an interface operation image collected in the previous image collection cycle.

[0073] The image similarity may represent the degree of similarity between the real-time operation image and the historical operation image. For example, the image similarity may be 20%, 50%, or 80%, etc.

[0074] Optionally, the determining the image similarity between the real-time operation image and the historical operation image includes:

[0075] Performing binary processing on the real-time operation image and the historical operation image respectively, to obtain a real-time binary image and a historical binary image in turn;

[0076] The image similarity between the real-time operation image and the historical operation image is determined according to the structural similarity index, the real-time binary image, and the historical binary image.

[0077] The real-time binary image may be understood as a binary image corresponding to the real-time operation image, and the historical binary image may be understood as a binary image corresponding to the historical operation image.

[0078] In the embodiment of the present invention, the specific manner of performing binarization processing on the operation image can be preset according to scene requirements and is not specifically limited here.

[0079] The structure similarity index (Structure Similarity Index Measure, SSIM) can be used to measure the similarity between two images.

[0080] S240: When the image similarity meets a preset condition, perform image verification on the real-time operation image through an interface verification model to obtain an image verification result.

[0081] Optionally, when the image similarity does not exceed a preset similarity threshold, the real-time operation image is verified by an interface verification model to obtain an image verification result.

[0082] In the embodiment of the present invention, the similarity threshold can be preset according to the scene requirements and is not specifically limited here. For example, the similarity threshold can be 70%, 80% or 90%.

[0083] S250: performing interface verification feedback on the human-computer interaction terminal based on the image verification result.

[0084] The technical solution of the embodiment of the present invention determines the image similarity between the real-time operation image and the historical operation image by determining the historical operation image corresponding to the real-time operation image; when the image similarity meets the preset conditions, the real-time operation image is image verified by the interface verification model. It is realized that only the interface operation image with a large operation change (change in operation content) compared with the operation image collected in the previous image collection cycle is verified, which reduces unnecessary image verification and saves computing costs.

[0085] Embodiment 3

[0086] Figure 3 This is a flowchart of a human-computer interaction operation interface verification method provided in Embodiment 3 of the present invention. This embodiment is an addition to the image verification of the real-time operation image by the interface verification model described in the above embodiment. Figure 3 As shown, the method includes:

[0087] S310: Acquire a training sample set through multiple image databases.

[0088] Each of the image databases is used to store interaction operation samples corresponding to the human-computer interaction terminal of an interaction terminal category.

[0089] The training sample set can be used to train an interface verification model.

[0090] The interactive operation samples may include historical interactive operation images and sample labels corresponding to the historical interactive operation images.

[0091] S320, training the convolutional neural network using the training sample set to obtain an interface verification model.

[0092] In an embodiment of the present invention, the convolutional neural network may be a deep learning model (Visual Geometry Group 16, VGG16) convolutional neural network for image recognition and classification.

[0093] S330, determining multiple human-computer interaction terminals to be interface verified.

[0094] S340: For each of the human-computer interaction terminals, collect a real-time operation image of the interactive operation interface of the human-computer interaction terminal.

[0095] S350: Perform image verification on the real-time operation image through an interface verification model to obtain an image verification result.

[0096] S360: Performing interface verification feedback on the human-computer interaction terminal based on the image verification result.

[0097] Optionally, after obtaining the image verification result, the method further includes:

[0098] Generate a new training sample based on the real-time operation image and the image verification result, and store the new training sample in an image database corresponding to the interactive terminal category of the human-computer interactive terminal corresponding to the real-time operation image;

[0099] Incremental training is performed on the interface verification model based on the newly added training samples to obtain an updated interface verification model.

[0100] Based on the above embodiment scheme, it is achieved that new training samples are generated based on the real-time operation image and the image verification result, and the interface verification model is incrementally trained based on the new training samples, thereby achieving further optimization of the interface verification model.

[0101] Furthermore, after obtaining the updated interface verification model, the real-time operation images collected periodically are verified using the updated interface verification model.

[0102] The technical solution of the embodiment of the present invention is to obtain a training sample set through multiple image databases, wherein each of the image databases is used to store interactive operation samples corresponding to the human-computer interactive end of an interactive end category; and to train a convolutional neural network through the training sample set to obtain an interface verification model. The interface verification model is obtained by training interactive operation samples of multiple interactive end categories, and the accuracy of the targeted verification of real-time operation images of different image categories (corresponding to the interactive end categories) by the trained interface verification model can be achieved.

[0103] Optionally, the overall process of the human-computer interaction operation interface verification method is exemplarily described as follows in the form of execution steps:

[0104] 1. Obtain image information of the human-computer interaction interface (i.e., the interactive operation interface) of each operating system, and perform data annotation according to the categories of different applied operating systems to determine the image category.

[0105] 1) Record the front-end interface of the operating system and take screenshots at a fixed frame rate to obtain the human-computer interaction interface during the interactive operation. The image contains information such as the human-computer interaction interface, processing information, feedback interface, and operation content.

[0106] 2) Creating workspaces (ie, the image database) according to the categories of different application operating systems, and determining a label for each workspace.

[0107] 3) Save the image in step 1) as a 3-channel (Red Green Blue, RGB) image in a Joint Photographic Group (JPG) format, perform pixel conversion on 1920 pixels * 1080 pixels, generate a static 2D (2Dimensions, 2D) image and store it in the corresponding working area.

[0108] 2. Correct the image information of the data set, enhance the data, and divide the data set.

[0109] 1) Based on the acquired image information, data enhancement (e.g., translation, rotation, mirroring, scaling, translation, and color rendering) is performed on each image, and the processed image is stored in the corresponding working range.

[0110] 2) All images are divided into training set and validation set in a ratio of 9:1.

[0111] 3) Assign the segmented data to different working intervals.

[0112] 3. Build a deep learning VGG16 convolutional neural network and train VGG16 through the training data set to obtain an interface verification model.

[0113] 1) The original 2D image of the human-computer interaction interface of the application operating system is compressed into an image of 244 pixels*244 pixels*3 channels.

[0114] 2) Perform convolution and pooling on the image, with a total of 13 convolutions and 5 poolings, using a maximum pooling layer to generate a feature map (characterizing the system operation interface targets at different sizes, positions, angles, brightness, etc.).

[0115] 3) After the fifth pooling, the data is converted from a multi-dimensional matrix into a one-dimensional vector through the Flatten layer to flatten the feature map into a feature vector, that is, to generate a vector of 1 pixel * 1 pixel * 1000 channels.

[0116] 4) Import the one-dimensional vector data into the activation layer and the fully connected layer, activate it using the rectified linear unit (ReLU), and complete the recognition and classification task.

[0117] 5) Based on the constructed neural network, images of training sets of application system operation records of different categories are imported and model training is performed.

[0118] 6) During model training, add callback functions to adjust parameters (e.g., learning rate) during training, and set functions such as automatic model saving and automatic training termination.

[0119] 7) Improve the recognition rate of the model by adjusting the parameters by comparing the verification results of the validation set. Adjust the model parameters according to the accuracy of the generated results on the validation set to optimize the neural network model.

[0120] 4. During the system interaction operation, the human-computer interaction interface is automatically captured and imported into the model to determine the system classification of the current operation interface.

[0121] 1) Automatically capture the human-computer interaction interface during the interactive operation at a period of 0.2 seconds, save it as incremental data in a temporary working area.

[0122] 2) When the human-computer interaction interface changes, image verification is performed again. Specifically, the previous image and the image input this time are binarized to obtain two binary images; the similarity between the two binary images is identified using the SSIM principle. If the identified binary data is different, that is, the similarity is low, it is considered that the human-computer interaction interface has changed.

[0123] 3) The above-mentioned changed images are compressed and imported into the interface verification model for classification and recognition.

[0124] 4) Image verification based on image category.

[0125] 5. Tesseract is an open source library for optical character recognition (OCR). It uses the image text extraction technology of the tesseract recognition engine to identify the information in the image (including operation results and processes), and judges whether the current interface meets the operation standards through the preset human-computer interaction judgment method.

[0126] 1) Use the tesseract recognition engine to extract text from the classified images of the human-computer interaction interface and save them.

[0127] 2) Preset standard keywords for operation execution feedback information as a comparison source, such as "succeed", "success", "completed", etc.

[0128] 3) If the human-computer interaction interface does not match the preset value of the comparison source, or the keyword is not searched, feedback is given to the interface to remind you to stop the current interaction operation.

[0129] 6. Feedback the judgment results to the interface and add the data to the original data set so that the interface verification model can perform incremental learning regularly.

[0130] Based on the technical solution of the present invention, the real-time verification of interactive operations is achieved by performing image verification on interface images using image recognition technology in visual technology, which can reduce the labor cost of interface verification and improve the timeliness of interface verification.

[0131] Embodiment 4

[0132] Figure 4 This is a schematic diagram of the structure of a human-computer interaction interface verification device provided by Embodiment 4 of the present invention. Figure 4 As shown, the device includes: an interactive end determination module 410, an image acquisition module 420, an image verification module 430 and a verification feedback module 440.

[0133] Among them, the interactive terminal determination module 410 is used to determine multiple human-computer interactive terminals to be interface verified; the image acquisition module 420 is used to collect the real-time operation image of the interactive operation interface of the human-computer interactive terminal for each of the human-computer interactive terminals, wherein the real-time operation image includes at least one of the operation target information, interactive operation information and interface feedback information; the image verification module 430 is used to perform image verification on the real-time operation image through an interface verification model to obtain an image verification result, wherein the image verification result represents the operation category of the human-computer interactive operation corresponding to the real-time operation image of the interactive operation interface, and the operation category includes normal operation and / or abnormal operation; the verification feedback module 440 is used to perform interface verification feedback on the human-computer interactive terminal based on the image verification result.

[0134] The technical solution of the embodiment of the present invention is to determine multiple human-computer interaction terminals to be interface verified; for each of the human-computer interaction terminals, collect the real-time operation image of the interactive operation interface of the human-computer interaction terminal, wherein the real-time operation image includes at least one of the operation target information, interactive operation information and interface feedback information; perform image verification on the real-time operation image through an interface verification model to obtain an image verification result, wherein the image verification result characterizes the operation category of the human-computer interaction operation corresponding to the real-time operation image of the interactive operation interface, and the operation category includes normal operation and / or abnormal operation; and perform interface verification feedback on the human-computer interaction terminal based on the image verification result. The present invention can achieve the effect of automatically performing real-time operation verification on the human-computer interaction operation interfaces of multiple human-computer interaction terminals at the same time. The present invention can reduce the waste of human resources and ensure the timeliness of verification, while ensuring the accuracy of the verification result.

[0135] Optionally, the image acquisition module 420 is specifically configured to:

[0136] An image acquisition cycle corresponding to the human-computer interaction terminal is determined, and a real-time operation image of the interactive operation interface of the human-computer interaction terminal is acquired according to the image acquisition cycle.

[0137] Optionally, the human-computer interaction operation interface verification device further includes:

[0138] a similarity calculation module, used to determine the historical operation image corresponding to the real-time operation image and determine the image similarity between the real-time operation image and the historical operation image before performing image verification on the real-time operation image through the interface verification model;

[0139] The image verification module 430 is specifically used for:

[0140] When the image similarity meets a preset condition, the real-time operation image is verified through an interface verification model.

[0141] Optionally, the similarity calculation module is specifically used to:

[0142] Performing binary processing on the real-time operation image and the historical operation image respectively, to obtain a real-time binary image and a historical binary image in turn;

[0143] The image similarity between the real-time operation image and the historical operation image is determined according to the structural similarity index, the real-time binary image, and the historical binary image.

[0144] Optionally, the image verification module 430 includes: an image category recognition unit and an image verification unit;

[0145] The image category recognition unit is used to determine the image category of the real-time operation image through an interface verification model, wherein the image category is related to the interactive terminal category of the human-computer interactive terminal corresponding to the real-time operation image;

[0146] The image verification unit is used to perform image verification on the real-time operation image based on the image category to obtain an image verification result.

[0147] Optionally, the image verification result includes verification passed and / or verification failed; the image verification unit is specifically used to:

[0148] Extracting text information from the real-time operation image to determine key text information;

[0149] In a case where the key text information satisfies a target text condition, the image verification result of the real-time operation image is determined to be the verification passed, wherein the target text condition is related to the image category of the real-time operation image.

[0150] Optionally, the human-computer interaction operation interface verification device further includes: a sample acquisition module and a model training module;

[0151] The sample acquisition module is used to acquire a training sample set through a plurality of image databases before performing image verification on the real-time operation image through the interface verification model, wherein each of the image databases is used to store interactive operation samples corresponding to the human-computer interactive terminal of an interactive terminal category;

[0152] The model training module is used to train the convolutional neural network through the training sample set to obtain an interface verification model.

[0153] Optionally, the human-computer interaction operation interface verification device further includes: a sample adding module and a model increment training module;

[0154] The sample adding module is used to generate a new training sample based on the real-time operation image and the image verification result after obtaining the image verification result, and store the new training sample in an image database corresponding to the interactive terminal category of the human-computer interaction terminal corresponding to the real-time operation image;

[0155] The model incremental training module is used to perform incremental training on the interface verification model based on the newly added training samples to obtain an updated interface verification model.

[0156] The human-computer interaction operation interface verification device provided in the embodiment of the present invention can execute the human-computer interaction operation interface verification method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0157] Embodiment 5

[0158] Figure 5 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0159] like Figure 5As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0160] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0161] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a human-computer interaction operation interface verification method.

[0162] In some embodiments, the human-computer interaction operation interface verification method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the human-computer interaction operation interface verification method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the human-computer interaction operation interface verification method in any other appropriate manner (e.g., by means of firmware).

[0163] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0164] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0165] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0166] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0167] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0168] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0169] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0170] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A human-computer interaction operation interface verification method, characterized in that: include: Determine multiple human-computer interaction terminals to be verified by the interface; For each of the human-computer interaction terminals, a real-time operation image of the interactive operation interface of the human-computer interaction terminal is collected, wherein the real-time operation image includes at least one of operation target information, interactive operation information, and interface feedback information; Performing image verification on the real-time operation image through an interface verification model to obtain an image verification result, wherein the image verification result represents an operation category of the human-computer interaction operation corresponding to the real-time operation image of the interactive operation interface, and the operation category includes normal operation and / or abnormal operation; Based on the image verification result, interface verification feedback is performed on the human-computer interaction terminal.

2. The method according to claim 1, characterized in that The collecting of the real-time operation image of the interactive operation interface of the human-computer interaction terminal includes: An image acquisition cycle corresponding to the human-computer interaction terminal is determined, and a real-time operation image of the interactive operation interface of the human-computer interaction terminal is acquired according to the image acquisition cycle.

3. The method according to claim 1, characterized in that Before performing image verification on the real-time operation image through the interface verification model, the method further includes: Determine a historical operation image corresponding to the real-time operation image, and determine an image similarity between the real-time operation image and the historical operation image; The performing image verification on the real-time operation image through the interface verification model includes: When the image similarity meets a preset condition, the real-time operation image is verified through an interface verification model.

4. The method according to claim 3, characterized in that The determining the image similarity between the real-time operation image and the historical operation image includes: Performing binary processing on the real-time operation image and the historical operation image respectively, to obtain a real-time binary image and a historical binary image in turn; The image similarity between the real-time operation image and the historical operation image is determined according to the structural similarity index, the real-time binary image, and the historical binary image.

5. The method according to claim 1, characterized in that The performing image verification on the real-time operation image through the interface verification model to obtain an image verification result includes: Determining an image category of the real-time operation image through an interface verification model, wherein the image category is related to an interaction terminal category of the human-computer interaction terminal corresponding to the real-time operation image; Performing image verification on the real-time operation image based on the image category to obtain an image verification result.

6. The method according to claim 5, characterized in that The image verification result includes verification pass and / or verification fail; the image verification is performed on the real-time operation image based on the image category to obtain the image verification result, including: Extracting text information from the real-time operation image to determine key text information; In a case where the key text information satisfies a target text condition, the image verification result of the real-time operation image is determined to be the verification passed, wherein the target text condition is related to the image category of the real-time operation image.

7. The method according to claim 1, characterized in that Before performing image verification on the real-time operation image through the interface verification model, the method further includes: Acquire a training sample set through a plurality of image databases, wherein each of the image databases is used to store interactive operation samples corresponding to the human-computer interactive terminal of an interactive terminal category; The convolutional neural network is trained using the training sample set to obtain an interface verification model.

8. The method according to claim 7, characterized in that After obtaining the image verification result, the method further includes: Generate a new training sample based on the real-time operation image and the image verification result, and store the new training sample in an image database corresponding to the interactive terminal category of the human-computer interactive terminal corresponding to the real-time operation image; Incremental training is performed on the interface verification model based on the newly added training samples to obtain an updated interface verification model.

9. A human-computer interaction interface verification device, characterized in that: include: An interaction terminal determination module, used to determine multiple human-computer interaction terminals to be interface verified; An image acquisition module, used for acquiring a real-time operation image of the interactive operation interface of each human-computer interaction terminal, wherein the real-time operation image includes at least one of operation target information, interactive operation information and interface feedback information; An image verification module, used to perform image verification on the real-time operation image through an interface verification model to obtain an image verification result, wherein the image verification result represents an operation category of the human-computer interaction operation corresponding to the real-time operation image of the interactive operation interface, and the operation category includes normal operation and / or abnormal operation; A verification feedback module is used to provide interface verification feedback to the human-computer interaction terminal based on the image verification result.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the human-computer interaction operation interface verification method according to any one of claims 1 to 8 when executed.

Images