Mouse operation event identification method and device, electronic equipment and readable medium

Abstract

Embodiments of the present disclosure disclose a mouse operation event recognition method and device, electronic equipment and readable medium. A specific embodiment of the method comprises: determining a frame image to be detected in a target video and a first reference frame image sequence and a second reference frame image sequence corresponding to the frame image to be detected; generating a first difference image sequence and a second difference image sequence according to the frame image to be detected, the first reference frame image sequence and the second reference frame image sequence; performing image superposition on the first reference frame image sequence, the frame image to be detected, the second reference frame image sequence, the first difference image sequence and the second difference image sequence to obtain a superimposed image; and inputting the superimposed image into a pre-trained mouse operation event recognition model to generate a mouse operation event corresponding to the frame image to be detected. The embodiment realizes effective and accurate recognition of a mouse operation event corresponding to a mouse contained in a video.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the field of computer technology, and more specifically to mouse operation event recognition methods, apparatus, electronic devices, and readable media. Background Technology

[0002] Mouse operation event recognition is a technology that identifies a series of mouse behaviors during computer operation. Currently, the common method for mouse operation event recognition is to record mouse operation events through the operating system's built-in mouse event listening interface.

[0003] However, the inventors discovered that the following technical problems often arise when using the above method:

[0004] First, the mouse event listening interface cannot identify the mouse operation events corresponding to the mouse contained in the video;

[0005] Second, the area of ​​change corresponding to the mouse in the video frame is often small. The method of directly calculating the difference between two frames to determine the mouse's change characteristics involves a large amount of data processing.

[0006] Third, mice often correspond to multiple operation types, and different operation types correspond to different operation events. Simply recording the states of mouse buttons or scroll wheels cannot effectively represent the specific operation events of the mouse, resulting in poor prediction accuracy of the trained model.

[0007] The information disclosed in this background section is only intended to enhance the understanding of the background of the inventive concept, and therefore may contain information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0008] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0009] Some embodiments of this disclosure provide methods, apparatus, electronic devices, and readable media for identifying mouse operation events to address one or more of the technical problems mentioned in the background section above.

[0010] In a first aspect, some embodiments of this disclosure provide a mouse operation event recognition method. The method includes: determining a frame image to be detected in a target video and a first reference frame image sequence and a second reference frame image sequence corresponding to the frame image to be detected, wherein the target video is a video representing a change in mouse state, the frame number of the first reference frame image is less than the frame number of the frame image to be detected, and the frame number of the second reference frame image is greater than the frame number of the frame image to be detected; generating a first difference image sequence and a second difference image sequence based on the frame image to be detected, the first reference frame image sequence, and the second reference frame image sequence, wherein the first difference image represents the image difference between the frame image to be detected and the first reference frame image, and the second difference image represents the image difference between the frame image to be detected and the second reference image; superimposing the first reference frame image sequence, the frame image to be detected, the second reference image sequence, the first difference image sequence, and the second difference image sequence to obtain a superimposed image; and inputting the superimposed image into a pre-trained mouse operation event recognition model to generate a mouse operation event corresponding to the frame image to be detected.

[0011] Secondly, some embodiments of this disclosure provide a mouse operation event recognition device, the device comprising: a determining unit configured to determine a frame image to be detected in a target video and a first reference frame image sequence and a second reference frame image sequence corresponding to the frame image to be detected, wherein the target video is a video characterizing a change in mouse state, the frame number of the first reference frame image is less than the frame number of the frame image to be detected, and the frame number of the second reference frame image is greater than the frame number of the frame image to be detected; and a generating unit configured to generate a first difference image based on the frame image to be detected, the first reference frame image sequence, and the second reference frame image sequence. The sequence and the second difference image sequence, wherein the first difference image represents the image difference between the frame image to be detected and the first reference frame image, and the second difference image represents the image difference between the frame image to be detected and the second reference frame image; the image overlay unit is configured to overlay the first reference frame image sequence, the frame image to be detected, the second reference frame image sequence, the first difference image sequence and the second difference image sequence to obtain an overlaid image; the input unit is configured to input the overlaid image into a pre-trained mouse operation event recognition model to generate a mouse operation event corresponding to the frame image to be detected.

[0012] Thirdly, some embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation of the first aspect above.

[0013] Fourthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method described in any of the implementations of the first aspect above.

[0014] The various embodiments of this disclosure have the following beneficial effects: the mouse operation event recognition method of some embodiments of this disclosure achieves effective and accurate recognition of mouse operation events corresponding to the mouse contained in a video. Specifically, the reason why mouse operation events corresponding to the mouse contained in a video cannot be recognized is that the mouse event listening interface cannot recognize mouse operation events corresponding to the mouse contained in the video. Based on this, the mouse operation event recognition method of some embodiments of this disclosure first determines the frame image to be detected in the target video and the first reference frame image sequence and the second reference frame image sequence corresponding to the frame image to be detected, wherein the target video is a video representing changes in mouse state, the frame number of the first reference frame image is less than the frame number of the frame image to be detected, and the frame number of the second reference frame image is greater than the frame number of the frame image to be detected. In practice, mouse operations often correspond to multiple frames, therefore, it is necessary to combine the images before and after the frame image to be detected to determine the mouse operation event corresponding to the frame image to be detected. Secondly, based on the aforementioned frame image to be detected, the first reference frame image sequence, and the second reference frame image sequence, a first difference image sequence and a second difference image sequence are generated. The first difference image represents the image difference between the frame image to be detected and the first reference frame image, and the second difference image represents the image difference between the frame image to be detected and the second reference frame image. By obtaining the first and second difference image sequences, the change features of the mouse cursor between different frames are obtained. Next, the first reference frame image sequence, the frame image to be detected, the second reference image sequence, the first difference image sequence, and the second difference image sequence are superimposed to obtain a superimposed image. Finally, the superimposed image is input into a pre-trained mouse operation event recognition model to generate the mouse operation event corresponding to the frame image to be detected. In this way, effective and accurate recognition of mouse operation events corresponding to the mouse cursor contained in the video is achieved. Attached Figure Description

[0015] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0016] Figure 1 This is a flowchart of some embodiments of the mouse operation event recognition method according to the present disclosure;

[0017] Figure 2 This is a schematic diagram showing the positional relationship between the frame image to be detected, the first reference frame image sequence, and the second reference frame image sequence.

[0018] Figure 3 This is a schematic diagram illustrating the generation process of the first and second difference image sequences;

[0019] Figure 4 This is a schematic diagram illustrating the generation process of the first difference image;

[0020] Figure 5 This is a schematic diagram illustrating the generation process of the first foreground and background images;

[0021] Figure 6 This is a schematic diagram of the structure of some embodiments of the mouse operation event recognition device according to the present disclosure;

[0022] Figure 7 This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0023] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0024] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0025] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0026] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0027] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0028] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0029] refer to Figure 1 The diagram illustrates a flow 100 of some embodiments of a mouse operation event recognition method according to the present disclosure. This mouse operation event recognition method includes the following steps:

[0030] Step 101: Determine the frame image to be detected in the target video and the corresponding first reference frame image sequence and second reference frame image sequence.

[0031] In some embodiments, the execution subject (e.g., a computing device) of the mouse operation event recognition method can determine the frame image to be detected in the target video and the corresponding first reference frame image sequence and second reference frame image sequence. The target video is a video representing changes in mouse state. The target video may contain multiple consecutive frames. In practice, the target video may be a video representing a series of mouse operation changes. The frame number of the first reference frame image may be less than the frame number of the frame image to be detected. The frame number of the second reference frame image may be greater than the frame number of the frame image to be detected. The frame number can represent the image position in the target video. In practice, the number of first reference frame images in the first reference frame image sequence can be K. The number of second reference frame images in the second reference frame image sequence can be M. Both K and M can be positive integers. In practice, the images included in the target video are all normalized, meaning the pixel values ​​corresponding to the pixels in the image range from [0,1]. Specifically, the target video includes RGB images, for example, the target video includes images with dimensions of 3×1280×720.

[0032] As an example, see Figure 2 The diagram illustrates the positional relationship between the frame image to be detected, the first reference frame image sequence, and the second reference frame image sequence. The target video 201 may contain multiple consecutive frames. The frame image to be detected 202 may be the 9th frame in the target video 201. The first reference frame image sequence 203 may contain two frames. Specifically, the first reference frame image sequence 203 may include the 7th and 8th frames in the target video 201. The second reference frame image sequence 204 may include eight frames. Specifically, the second reference frame image sequence 204 may include frames 10 through 17 in the target video 201.

[0033] It should be noted that the aforementioned computing devices can be either hardware or software. When the computing device is hardware, it can be implemented as a distributed cluster consisting of multiple servers or terminal devices, or as a single server or terminal device. When the computing device is software, it can be installed on the hardware devices listed above. It can be implemented as, for example, multiple software programs or software modules used to provide distributed services, or as a single software program or software module. No specific limitations are made here. It should be understood that the number of computing devices can be arbitrary, depending on the implementation requirements.

[0034] Step 102: Generate a first difference image sequence and a second difference image sequence based on the image to be detected, the first reference frame image sequence, and the second reference frame image sequence.

[0035] In some embodiments, the execution entity can generate a first difference image sequence and a second difference image sequence based on the frame image to be detected, a first reference frame image sequence, and a second reference frame image sequence. The first difference image represents the image difference between the frame image to be detected and the first reference frame image. The second difference image represents the image difference between the frame image to be detected and the second reference frame image. In practice, the execution entity can determine the first difference image or the second difference image by subtracting the images.

[0036] As an example, such as Figure 3 The diagram illustrates the generation process of the first and second difference image sequences. First, the executing entity determines the image difference between each first reference frame image in the frame image to be detected 202 and the first reference frame image sequence 203, using this as the first difference image to obtain the first difference image sequence 301. Second, the executing entity determines the image difference between each second reference frame image in the second reference frame image sequence 204 and the frame image to be detected 202, using this as the second difference image to obtain the second difference image sequence 302.

[0037] In some optional implementations of certain embodiments, the execution entity may generate a first difference image sequence and a second difference image sequence based on the frame image to be detected, a first reference frame image sequence, and a second reference frame image sequence, which may include the following steps:

[0038] First, for each first reference frame image in the above first reference frame image sequence, perform the following first processing step:

[0039] The first sub-step involves determining the image difference between the sub-image of the first reference frame image under the first color channel and the sub-image of the frame image to be detected under the first color channel, thereby obtaining a first sub-difference image.

[0040] The first color channel mentioned above can be the red channel in the RGB color channel.

[0041] The second sub-step involves determining the image difference between the sub-image of the first reference frame image in the second color channel and the sub-image of the frame image to be detected in the second color channel, thereby obtaining a second sub-difference image.

[0042] The second color channel mentioned above can be the green channel in the RGB color channel.

[0043] The third sub-step involves determining the image difference between the sub-image of the first reference frame image in the third color channel and the sub-image of the frame image to be detected in the third color channel, thereby obtaining the third sub-difference image.

[0044] The third color channel mentioned above is the blue channel in the RGB color channel.

[0045] The fourth sub-step involves generating a first difference image corresponding to the first reference frame image in the first difference image sequence based on the first sub-difference image, the second sub-difference image, and the third sub-difference image.

[0046] In practice, the aforementioned executing entity can overlay the first sub-difference image, the second sub-difference image, and the third sub-difference image to generate the first difference image corresponding to the first reference frame image in the first difference image sequence.

[0047] As an example, such as Figure 4 The diagram illustrates the generation process of the first difference image. For the first reference frame image 401, firstly, the executing entity determines the image difference between the sub-image 4011 of the first reference frame image 401 in the first color channel and the sub-image of the frame image 202 to be detected in the first color channel, obtaining a first sub-difference image 4021. Then, the executing entity determines the image difference between the sub-image 4012 of the first reference frame image 401 in the second color channel and the sub-image of the frame image 202 to be detected in the second color channel, obtaining a second sub-difference image 4022. Next, the executing entity determines the image difference between the sub-image 4013 of the first reference frame image 401 in the third color channel and the sub-image of the frame image 202 to be detected in the third color channel, obtaining a third sub-difference image 4023. Finally, the aforementioned execution entity can overlay the first sub-difference image 4021, the second sub-difference image 4022, and the third sub-difference image 4023 to generate the first difference image 403 corresponding to the first reference frame image 401 in the first difference image sequence.

[0048] The second step involves performing the following second processing step for each second reference frame image in the aforementioned second reference frame image sequence:

[0049] The first sub-step involves determining the image difference between the sub-image of the second reference frame image under the first color channel and the sub-image of the frame to be detected under the first color channel, thereby obtaining a fourth sub-difference image.

[0050] The first color channel mentioned above can be the red channel in the RGB color channel.

[0051] The second sub-step involves determining the image difference between the sub-image of the second reference frame image in the second color channel and the sub-image of the frame image to be detected in the second color channel, thus obtaining the fifth sub-difference image.

[0052] The second color channel mentioned above can be the green channel in the RGB color channel.

[0053] The third sub-step involves determining the image difference between the sub-image of the second reference frame image in the third color channel and the sub-image of the frame image to be detected in the third color channel, thus obtaining the sixth sub-difference image.

[0054] The third color channel mentioned above is the blue channel in the RGB color channel.

[0055] The fourth sub-step involves generating a second difference image corresponding to the second reference frame image in the second difference image sequence based on the fourth, fifth, and sixth sub-difference images.

[0056] In some optional implementations of certain embodiments, generating the first difference image sequence and the second difference image sequence based on the image to be detected, the first reference frame image sequence, and the second reference frame image sequence may include the following steps:

[0057] For each first reference frame image in the above first reference frame image sequence, the following first processing step is performed:

[0058] The first step is to input the above-mentioned frame image to be detected and the above-mentioned first reference frame image into a pre-trained image change region detection model to generate first change region information and second change region information.

[0059] The first change region information represents the mouse position in the frame image to be detected. The second change region information represents the mouse position in the first reference frame image. In practice, the first change region information represents the bounding box of interest corresponding to the mouse position in the frame image to be detected. The second change region information represents the bounding box of interest corresponding to the mouse position in the first reference frame image.

[0060] Specifically, an image change region detection model can include: a first feature extraction model, a second feature extraction model, a feature fusion layer, and a change region detection model. The first and second feature extraction models have the same model structure. In practice, the first feature extraction model can be a U-Net model. The feature fusion layer is used to superimpose the outputs of the first and second feature extraction models. The change region detection model can be a MobileNet_YOLOv3 model, which is a lightweight detection model based on the YOLO model.

[0061] The second step involves cropping the first reference frame image based on the first change region information and the second change region information to obtain a first foreground image and a background image.

[0062] The first foreground image represents the mouse position in the first reference frame image corresponding to the first and second change region information. Since the image size of the first reference frame image is the same as the image size of the frame image to be detected, the position corresponding to the first change region information can be projected onto the first reference frame image. The background image represents the image in the first reference frame image excluding the first foreground image.

[0063] As an example, such as Figure 5 The diagram illustrates the generation process of the first foreground and background images. It shows the position of the bounding box 502 corresponding to the first change region information in the first reference frame image 501, and the position of the bounding box 503 corresponding to the second change region information in the first reference frame image 501. It also shows the first foreground image 505 and the background image 504 obtained by cropping the first reference frame image 501.

[0064] The third step involves cropping the image of the frame to be detected based on the first change region information and the second change region information to obtain the second foreground image.

[0065] The second foreground image represents the position of the mouse corresponding to the first and second change region information in the frame image to be detected. The generation method of the second foreground image can refer to the generation method of the first foreground image, that is, by projection, the position corresponding to the second change region information is projected onto the frame image to be detected and then cropped.

[0066] The fourth step is to determine the image differences between the second foreground image and the first foreground image to obtain the target difference image.

[0067] In practice, the target difference image = second foreground image - first foreground image.

[0068] The fifth step is to overlay the target difference image and the background image to obtain the first difference image corresponding to the first reference frame image in the first difference image sequence.

[0069] The first to fifth steps described above constitute an inventive point of this disclosure, solving the second technical problem mentioned in the background art, namely, "the changing area corresponding to the mouse in a video frame often occupies a small proportion, and the method of directly calculating the difference between two frames to determine the mouse's changing characteristics results in a large amount of data processing." In practice, for videos containing multiple consecutive frames, the method of directly calculating the difference between two frames leads to an extremely large amount of data processing as the number of frames increases and the image size is affected. Based on this, this disclosure designs an image changing region detection model. First, features are extracted from the first reference frame image and the frame image to be detected using the first feature extraction model and the second feature extraction model included in the image changing region detection model. Second, considering that the mouse occupies a small proportion in the image, and in order to improve the detection speed, a lightweight changing region detection model is designed to perform the final changing region detection to obtain the first changing region information and the second changing region information. Finally, considering that the purpose of determining the image differences between images is mainly to determine the mouse's changing characteristics. Therefore, this disclosure performs image cropping on the first reference frame image based on the first change region information and the second change region information to obtain a first foreground image and a background image; performs image cropping on the frame image to be detected based on the first change region information and the second change region information to obtain a second foreground image; determines the image difference between the second foreground image and the first foreground image to obtain a target difference image; and superimposes the target difference image and the background image to obtain the first difference image corresponding to the first reference frame image in the first difference image sequence. In this way, only the image difference at the mouse location is needed, which greatly reduces the computational load, especially when determining the image differences of multiple frames.

[0070] Step 103: Overlay the first reference frame image sequence, the frame image to be detected, the second reference frame image sequence, the first difference image sequence, and the second difference image sequence to obtain the overlaid image.

[0071] In some embodiments, the execution entity may overlay the first reference frame image sequence, the frame image to be detected, the second reference frame image sequence, the first difference image sequence, and the second difference image sequence to obtain the overlaid image.

[0072] In some optional implementations of certain embodiments, the execution entity performs image overlay on the first reference frame image sequence, the frame image to be detected, the second reference frame image sequence, the first difference image sequence, and the second difference image sequence to obtain the overlaid image, which may include the following steps:

[0073] The first step is to overlay the first reference frame image sequence, the frame image to be detected, and the second reference frame image sequence according to the order of their corresponding frame numbers to obtain the first overlaid image.

[0074] As an example, the frame numbers of the first reference frame images in the first reference frame image sequence can be frame 7 and frame 8, respectively. The frame number of the frame image to be detected can be frame 9. Specifically, firstly, the above execution can superimpose the first reference frame image of frame 7 onto the first reference frame image of frame 8. Secondly, the above execution body can superimpose the superimposed first reference frame images of frames 7 and 8 onto the frame image to be detected of frame 9.

[0075] The second step involves superimposing the first difference image sequence and the second difference image sequence according to the frame number of the corresponding first reference frame image or the frame number of the corresponding second reference frame image to obtain the second superimposed image.

[0076] The third step is to overlay the first superimposed image and the second superimposed image to obtain the superimposed image.

[0077] As an example, the aforementioned executing entity can overlay the second superimposed image onto the first superimposed image to obtain the superimposed image.

[0078] Step 104: Input the superimposed image into the pre-trained mouse operation event recognition model to generate the mouse operation event corresponding to the frame image to be detected.

[0079] In some embodiments, the aforementioned execution entity can input the superimposed image into a pre-trained mouse operation event recognition model to generate mouse operation events corresponding to the frame image to be detected. The mouse operation event recognition model can be a model used to recognize mouse operation events. In practice, the mouse operation event recognition model can be a CNN (Convolutional Neural Networks) model. Mouse operation events can represent mouse operation events. Specifically, mouse operation events can include, but are not limited to, any of the following: left mouse button still event, left mouse button click event, left mouse button double click event, left mouse button drag event, middle mouse button still event, middle mouse button click event, middle mouse button double click event, right mouse button still event, right mouse button right click event, right mouse button double click event, right mouse button drag event, mouse wheel still event, mouse wheel scrolling event, and mouse wheel scrolling event. For example, the output of the mouse operation event recognition model can be a 1×15 dimensional vector. The aforementioned execution entity can map the mouse operation events corresponding to the frame image to be detected based on the output of the mouse operation event recognition model.

[0080] Optionally, the mouse operation event recognition model described above may include: a feature extraction model and a multi-head perceptron model. The feature extraction model is used to extract features from the superimposed image. In practice, the feature extraction model can be a ResNet model. Alternatively, it can be a VGGNet model. The multi-head perceptron model can be a Multilayer Perceptron (MLP) model. In practice, the multi-head perceptron model may include: a first perceptron model, a second perceptron model, a third perceptron model, and a fourth perceptron model. The first perceptron model is used to predict mouse operation events for the left mouse button. The second perceptron model is used to predict mouse operation events for the middle mouse button. The third perceptron model is used to predict mouse operation events for the right mouse button. The fourth perceptron model is used to predict mouse operation events for the mouse wheel. Specifically, the first, second, third, and fourth perceptron models can all be MLP models. Specifically, the MLP model may include: at least one linear layer and a nonlinear layer. The linear layer includes at least one neuron. The nonlinear layer may correspond to an activation function. For example, the activation function could be the Sigmoid function. Verification showed that the single-layer perceptron model and the multi-layer perceptron model exhibited little difference in prediction performance for mouse operation events.

[0081] In some optional implementations of certain embodiments, the execution entity inputs the superimposed image into a pre-trained mouse operation event recognition model to generate the mouse operation event corresponding to the frame image to be detected, which may include the following steps:

[0082] The first step is to extract features from the superimposed image using the aforementioned feature extraction model to generate extracted image features.

[0083] The second step is to input the extracted image features into the multi-head mechanism-based perception model to generate mouse operation event vectors.

[0084] In practice, the mouse operation event vector described above is a 1×15 dimensional vector. Specifically, the vector values ​​in the mouse operation event vector correspond to mouse operation events. For example, the first dimension of the mouse operation event vector corresponds to the left mouse button still event.

[0085] Optionally, the mouse operation event recognition model can be trained through the following steps:

[0086] The first step is to obtain the video to be labeled.

[0087] The aforementioned video to be labeled can be a video that includes changes in mouse state.

[0088] In practice, the aforementioned implementing entities can obtain the videos to be annotated via wired or wireless connections.

[0089] The second step involves performing the following state transition steps for each frame of the video to be labeled:

[0090] The first sub-step is to determine the type of mouse operation object corresponding to the above-mentioned frame image to be labeled.

[0091] The mouse operation object type represents the mouse operation objects included in the frame image to be labeled. In practice, operation objects can include mouse buttons and mouse wheels. Specifically, mouse buttons can include: the left mouse button, the middle mouse button, and the right mouse button. For example, the mouse operation objects annotated by the annotator for the image to be labeled can be identified as the mouse operation object type.

[0092] The second sub-step is to determine the mouse operation object type as the mouse button operation type and the mouse included in the frame image to be labeled satisfies the first condition, and to determine the mouse drag event as the image sample label corresponding to the frame image to be labeled.

[0093] The first condition is that the mouse in the frame image to be labeled is in a moving state and in a pressed state.

[0094] The third sub-step is to determine the mouse operation object type as a mouse button operation type and the mouse included in the frame image to be labeled satisfies the second condition, and to determine the mouse double-click event as the image sample label corresponding to the frame image to be labeled.

[0095] The second condition is that the time interval between two consecutive mouse press states in the frame image to be labeled is less than a preset time interval.

[0096] The fourth sub-step is to determine the mouse operation object type as a mouse button operation type and the mouse included in the frame image to be labeled satisfies the third condition, and to determine the mouse click event as the image sample label corresponding to the frame image to be labeled.

[0097] The third condition is that the frame image to be labeled includes the mouse cursor ending its press state.

[0098] The fifth sub-step, in response to determining that the mouse operation object type is a mouse scroll wheel operation type, determines the mouse state of the mouse included in the frame image to be labeled as the image sample label corresponding to the frame image to be labeled.

[0099] In practice, when the above-mentioned mouse operation object type is the mouse wheel operation type, the mouse state can be the mouse wheel stationary state, the mouse wheel scrolling upwards state, or the mouse wheel scrolling downwards state.

[0100] Specifically, the first to fifth sub-steps can be implemented by constructing a state transition machine.

[0101] The third step involves supervising the initial mouse operation event recognition model based on the video to be labeled and the corresponding image sample label sequence, thereby obtaining the mouse operation event recognition model.

[0102] The first to third steps described above constitute an inventive point of this disclosure, solving the third technical problem mentioned in the background art, namely, "a mouse often corresponds to multiple operation types, and different operation types correspond to different operation events. Simply recording the states of mouse buttons or scroll wheels cannot effectively represent the specific operation events of the mouse, resulting in poor prediction accuracy of the trained model." In practice, a mouse often includes buttons and a scroll wheel. Furthermore, buttons can be further subdivided into left, right, and middle buttons. Therefore, a mouse can correspond to multiple operation types. To improve the prediction accuracy of the model, this disclosure designs a state transition machine to execute the first to fifth sub-steps in the second step described above. By combining the mouse operation object type, the mouse pressing state, and the movement state, the corresponding mouse operation events are automatically subdivided to obtain corresponding image sample labels. The training samples obtained in this way can improve the prediction accuracy of the trained mouse operation event recognition model.

[0103] Optionally, for each frame of the video to be labeled, the execution entity can directly determine the mouse operation object type included in the frame as the image sample label corresponding to the frame. In this case, the mouse operation event recognition model obtained through supervised training based on the video and the corresponding image sample label sequence can be used to identify the state of the mouse operation object contained in the image. Specifically, the output of the mouse operation event recognition model is a 1×5-dimensional vector. For example, the first vector value in the 1×5-dimensional vector output by the mouse operation event recognition model can represent the left mouse button, the second vector value can represent the middle mouse button, the third vector value can represent the right mouse button, the fourth vector value can represent scrolling the mouse wheel, and the fifth vector value can represent scrolling the mouse wheel down.

[0104] The various embodiments of this disclosure have the following beneficial effects: the mouse operation event recognition method of some embodiments of this disclosure achieves effective and accurate recognition of mouse operation events corresponding to the mouse contained in a video. Specifically, the reason why mouse operation events corresponding to the mouse contained in a video cannot be recognized is that the mouse event listening interface cannot recognize mouse operation events corresponding to the mouse contained in the video. Based on this, the mouse operation event recognition method of some embodiments of this disclosure first determines the frame image to be detected in the target video and the first reference frame image sequence and the second reference frame image sequence corresponding to the frame image to be detected, wherein the target video is a video representing changes in mouse state, the frame number of the first reference frame image is less than the frame number of the frame image to be detected, and the frame number of the second reference frame image is greater than the frame number of the frame image to be detected. In practice, mouse operations often correspond to multiple frames, therefore, it is necessary to combine the images before and after the frame image to be detected to determine the mouse operation event corresponding to the frame image to be detected. Secondly, based on the aforementioned frame image to be detected, the first reference frame image sequence, and the second reference frame image sequence, a first difference image sequence and a second difference image sequence are generated. The first difference image represents the image difference between the frame image to be detected and the first reference frame image, and the second difference image represents the image difference between the frame image to be detected and the second reference frame image. By obtaining the first and second difference image sequences, the change features of the mouse cursor between different frames are obtained. Next, the first reference frame image sequence, the frame image to be detected, the second reference image sequence, the first difference image sequence, and the second difference image sequence are superimposed to obtain a superimposed image. Finally, the superimposed image is input into a pre-trained mouse operation event recognition model to generate the mouse operation event corresponding to the frame image to be detected. In this way, effective and accurate recognition of mouse operation events corresponding to the mouse cursor contained in the video is achieved.

[0105] Further reference Figure 6 As an implementation of the methods shown in the above figures, this disclosure provides some embodiments of a mouse operation event recognition device, which are similar to... Figure 1 Corresponding to the method embodiments shown, this mouse operation event recognition device can be specifically applied to various electronic devices.

[0106] like Figure 6 As shown, a mouse operation event recognition device 600 in some embodiments includes: a determining unit 601, a generating unit 602, an image overlay unit 603, and an input unit 604. The determining unit 601 is configured to determine a frame image to be detected in a target video and a first reference frame image sequence and a second reference frame image sequence corresponding to the frame image to be detected. The target video is a video representing a change in mouse state. The frame number of the first reference frame image is less than the frame number of the frame image to be detected, and the frame number of the second reference frame image is greater than the frame number of the frame image to be detected. The generating unit 602 is configured to generate a first difference image sequence and a second difference image sequence based on the frame image to be detected, the first reference frame image sequence, and the second reference frame image sequence. The first difference image represents the image difference between the frame image to be detected and the first reference frame image, and the second difference image represents the image difference between the frame image to be detected and the second reference frame image; the image overlay unit 603 is configured to overlay the first reference frame image sequence, the frame image to be detected, the second reference frame image sequence, the first difference image sequence, and the second difference image sequence to obtain an overlaid image; the input unit 604 is configured to input the overlaid image into a pre-trained mouse operation event recognition model to generate a mouse operation event corresponding to the frame image to be detected.

[0107] It is understandable that the units described in the mouse operation event recognition device 600 are related to the reference... Figure 1 The steps in the described method correspond to each other. Therefore, the operations, features, and beneficial effects described above for the method also apply to the mouse operation event recognition device 600 and the units contained therein, and will not be repeated here.

[0108] The following is for reference. Figure 7 It shows a schematic diagram of the structure of an electronic device (e.g., a computing device) 700 suitable for implementing some embodiments of the present disclosure. Figure 7 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0109] like Figure 7As shown, the electronic device 700 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory 702 or a program loaded from a storage device 708 into a random access memory 703. The random access memory 703 also stores various programs and data required for the operation of the electronic device 700. The processing unit 701, the read-only memory 702, and the random access memory 703 are interconnected via a bus 704. An input / output interface 705 is also connected to the bus 704.

[0110] Typically, the following devices can be connected to I / O interface 705: input devices 706 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 707 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 708 including, for example, magnetic tapes, hard disks, etc.; and communication devices 709. Communication device 709 allows electronic device 700 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 7 An electronic device 700 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 7 Each box shown can represent a device or multiple devices as needed.

[0111] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 709, or installed from a storage device 708, or installed from a read-only memory 702. When the computer program is executed by the processing device 701, it performs the functions defined in the methods of some embodiments of this disclosure.

[0112] It should be noted that, in some embodiments of this disclosure, the computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0113] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed networks.

[0114] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs. When the electronic device executes the aforementioned one or more programs, the electronic device causes the following to occur: determine a frame image to be detected in a target video and a first reference frame image sequence and a second reference frame image sequence corresponding to the frame image to be detected, wherein the target video is a video representing a change in mouse state, the frame number of the first reference frame image is less than the frame number of the frame image to be detected, and the frame number of the second reference frame image is greater than the frame number of the frame image to be detected; generate a first difference image sequence and a second difference image sequence based on the frame image to be detected, the first reference frame image sequence, and the second reference frame image sequence, wherein the first difference image represents the image difference between the frame image to be detected and the first reference frame image, and the second difference image represents the image difference between the frame image to be detected and the second reference image; overlay the first reference frame image sequence, the frame image to be detected, the second reference image sequence, the first difference image sequence, and the second difference image sequence to obtain an overlaid image; and input the overlaid image into a pre-trained mouse operation event recognition model to generate a mouse operation event corresponding to the frame image to be detected.

[0115] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0116] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0117] The units described in some embodiments of this disclosure can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor may be described as including a determining unit, a generating unit, an image overlay unit, and an input unit. The names of these units do not necessarily limit the specific unit itself; for example, the image overlay unit may also be described as "a unit that overlays the first reference frame image sequence, the frame image to be detected, the second reference frame image sequence, the first difference image sequence, and the second difference image sequence to obtain an overlaid image."

[0118] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0119] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A mouse operation event identification method, comprising: determining a to-be-detected frame image in a target video and a first reference frame image sequence and a second reference frame image sequence corresponding to the to-be-detected frame image, wherein the target video is a video representing a mouse state change, a frame sequence number of a first reference frame image is less than a frame sequence number of the to-be-detected frame image, and a frame sequence number of a second reference frame image is greater than the frame sequence number of the to-be-detected frame image; generating a first difference image sequence and a second difference image sequence according to the to-be-detected frame image, the first reference frame image sequence, and the second reference frame image sequence, wherein a first difference image represents an image difference between the to-be-detected frame image and a first reference frame image, and a second difference image represents an image difference between the to-be-detected frame image and a second reference frame image, wherein for each first reference frame image in the first reference frame image sequence, the following first processing steps are performed: inputting the to-be-detected frame image and the first reference frame image into a pre-trained image change region detection model to generate first change region information and second change region information; performing image cropping on the first reference frame image according to the first change region information and the second change region information to obtain a first foreground image and a background image; performing image cropping on the to-be-detected frame image according to the first change region information and the second change region information to obtain a second foreground image; determining an image difference between the second foreground image and the first foreground image to obtain a target difference image; and superimposing the target difference image and the background image to obtain a first difference image corresponding to the first reference frame image in the first difference image sequence; performing image superimposition on the first reference frame image sequence, the to-be-detected frame image, the second reference frame image sequence, the first difference image sequence, and the second difference image sequence according to a sequence order of corresponding frame sequence numbers to obtain a superimposed image, comprising: performing image superimposition on the first reference frame image sequence, the to-be-detected frame image, and the second reference frame image sequence according to the sequence order of the corresponding frame sequence numbers to obtain a first superimposed image; performing image superimposition on the first difference image sequence and the second difference image sequence according to a frame sequence number of a corresponding first reference frame image or a frame sequence number of a corresponding second reference frame image to obtain a second superimposed image; and performing image superimposition on the first superimposed image and the second superimposed image to obtain the superimposed image; inputting the superimposed image into a pre-trained mouse operation event identification model to generate a mouse operation event corresponding to the to-be-detected frame image, wherein the mouse operation event identification model comprises a feature extraction model and a perception model based on a multi-head mechanism; and the inputting the superimposed image into the pre-trained mouse operation event identification model to generate the mouse operation event corresponding to the to-be-detected frame image comprises: performing feature extraction on the superimposed image through the feature extraction model to generate extracted image features. input the extracted image features into the perception model based on the multi-head mechanism to generate a mouse operation event vector; generate the mouse operation event according to the mouse operation event vector.

2. A mouse operation event recognition device, comprising: a determination unit configured to determine a frame image to be detected in a target video and a first reference frame image sequence and a second reference frame image sequence corresponding to the frame image to be detected, wherein the target video is a video representing a change in a mouse state, the frame sequence number of the first reference frame image is less than the frame sequence number of the frame image to be detected, and the frame sequence number of the second reference frame image is greater than the frame sequence number of the frame image to be detected; a generation unit configured to generate a first difference image sequence and a second difference image sequence according to the frame image to be detected, the first reference frame image sequence, and the second reference frame image sequence, wherein a first difference image represents an image difference between the frame image to be detected and a first reference frame image, and a second difference image represents an image difference between the frame image to be detected and a second reference frame image, and for each first reference frame image in the first reference frame image sequence, the following first processing steps are performed: inputting the frame image to be detected and the first reference frame image into a pre-trained image change region detection model to generate first change region information and second change region information; performing image cropping on the first reference frame image according to the first change region information and the second change region information to obtain a first foreground image and a background image; performing image cropping on the frame image to be detected according to the first change region information and the second change region information to obtain a second foreground image; determining an image difference between the second foreground image and the first foreground image to obtain a target difference image; and superimposing the target difference image and the background image to obtain a first difference image corresponding to the first reference frame image in the first difference image sequence; an image superimposition unit configured to superimpose the first reference frame image sequence, the frame image to be detected, the second reference frame image sequence, the first difference image sequence, and the second difference image sequence according to the order of corresponding frame sequence numbers to obtain a superimposed image; an input unit configured to input the superimposed image into a pre-trained mouse operation event recognition model to generate a mouse operation event corresponding to the frame image to be detected, wherein the mouse operation event recognition model comprises a feature extraction model and a perception model based on a multi-head mechanism; the input unit is further configured to: extract features of the superimposed image through the feature extraction model to generate extracted image features; input the extracted image features into the perception model based on the multi-head mechanism to generate a mouse operation event vector; generate the mouse operation event according to the mouse operation event vector; wherein the image superimposition unit is further configured to: According to the order of the corresponding frame numbers, the first reference frame image sequence, the frame image to be detected and the second reference frame image sequence are superimposed to obtain a first superimposed image; According to the frame number of the corresponding first reference frame image or the frame number of the corresponding second reference frame image, the first difference image sequence and the second difference image sequence are superimposed to obtain a second superimposed image; The first superimposed image and the second superimposed image are superimposed to obtain the superimposed image. 3.An electronic device, comprising: one or more processors; a storage having stored thereon one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the method of claim 1.

4. A computer readable medium having stored thereon a computer program, wherein, The computer program is executed by the processor to implement the method of claim 1.

Images