Action recognition method and device, electronic device and storage medium

Abstract

The invention relates to an action recognition method and device, an electronic device and a storage medium, and the method comprises the steps: obtaining a to-be-detected video segment; inputting a to-be-detected video segment into the trained target action recognition model for processing; and utilizing a multi-head attention mechanism network in the target action recognition model to fuse the features of each frame of image in the to-be-detected video segment to obtain a fused feature, and outputting an action recognition result according to the fused feature. Through the method and the device, the problem of low accuracy of sewing action recognition is solved.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a method, device, electronic device and storage medium for motion recognition. Background Art

[0002] In the current medical field, effective recognition and evaluation of intraoperative suturing actions are of great significance. In related technologies, manual recognition and other methods have problems such as high subjectivity and long time consumption, resulting in low accuracy in intraoperative suturing action recognition, which is difficult to meet actual clinical needs.

[0003] Currently, no effective solution has been proposed for the problem of low accuracy in sewing action recognition in related technologies. Summary of the invention

[0004] The embodiments of the present application provide a method, device, electronic device and storage medium for motion recognition, so as to at least solve the problem of low accuracy of stitching motion recognition in the related art.

[0005] In a first aspect, an embodiment of the present application provides a method for action recognition, the method comprising:

[0006] Get the video segment to be tested;

[0007] The video segment to be tested is input into the trained target action recognition model for processing; the multi-head attention mechanism network in the target action recognition model is used to fuse the image features of each frame in the video segment to be tested to obtain a fused feature, and the action recognition result is output according to the fused feature.

[0008] In some embodiments, the multi-head attention mechanism network in the target action recognition model is used to fuse the image features of each frame in the video segment to be tested to obtain the fused features, including:

[0009] Inputting each frame image in the video segment to be tested into the feature extraction network in the target action recognition model, and outputting the image features corresponding to each frame image;

[0010] Input each of the image features in parallel into the multi-head attention mechanism network to obtain the fused features.

[0011] In some embodiments, obtaining the video segment to be tested includes:

[0012] Acquire an image to be tested; perform bright channel processing on the image to be tested to obtain a target image;

[0013] Acquire a plurality of frames of images collected before and / or after the target image; and generate the video segment to be tested based on the plurality of frames of images and the target image.

[0014] In some embodiments, performing bright channel processing on the image to be tested to obtain a target image includes:

[0015] Performing bright channel processing on each pixel of the image to be tested to determine the maximum pixel value in a preset color channel; and generating a grayscale image based on the maximum pixel value;

[0016] Perform maximum value filtering on the grayscale image to obtain the target image.

[0017] In some embodiments, performing bright channel processing on the image to be tested to obtain a target image includes:

[0018] Performing bright channel processing on the image to be tested to obtain a bright channel image; performing normalization processing based on the bright channel image to obtain a weight matrix;

[0019] Acquire the cumulative distribution function of the image to be tested; perform weighted equalization processing on the image to be tested according to the weight matrix and the cumulative distribution function to obtain the target image.

[0020] In some embodiments, the multi-head attention mechanism network in the target action recognition model is used to fuse the image features of each frame in the video segment to be tested to obtain the fused features, including:

[0021] The multi-head attention mechanism network is used to weightedly fuse the image features corresponding to the target image with the image features corresponding to other images to obtain the fused features; the other images are images other than the target image in the video segment to be tested.

[0022] In some embodiments, training the target action recognition model includes:

[0023] Obtaining a training image set; the training image set carries multiple category labels;

[0024] Inputting the training image set into an initial action recognition model including the multi-head attention mechanism network for training, and outputting a predicted category result;

[0025] A loss function is calculated according to the predicted category result and the category label; and the gradient of the loss function is back-transferred to the initial action recognition model for iteration to generate the target action recognition model.

[0026] In a second aspect, an embodiment of the present application provides a motion recognition device, the device comprising:

[0027] An acquisition module, used to acquire the video segment to be tested;

[0028] The recognition module is used to input the video segment to be tested into the trained target action recognition model for processing; utilize the multi-head attention mechanism network in the target action recognition model to fuse the image features of each frame in the video segment to be tested, obtain the fused features, and output the action recognition result according to the fused features.

[0029] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the motion recognition method as described in the first aspect above is implemented.

[0030] In a fourth aspect, an embodiment of the present application provides a storage medium on which a computer program is stored, and when the program is executed by a processor, the action recognition method described in the first aspect above is implemented.

[0031] Compared with the related art, the action recognition method, device, electronic device and storage medium provided in the embodiment of the present application obtain the video segment to be tested; input the video segment to be tested into the trained target action recognition model for processing; use the multi-head attention mechanism network in the target action recognition model to fuse the image features of each frame in the video segment to be tested, obtain the fusion feature, and output the action recognition result according to the fusion feature. Based on this, the model can be enabled to have the ability to pay attention to the pictures at different times at the same time, obtain more comprehensive and accurate information, and in the forward process, multiple attention heads are processed in parallel, so that the model can model the input image from multiple angles and better capture the relationship between multiple frames, so that the weights of the mutual correlation between frames can be reasonably allocated to achieve the purpose of better identifying fine movements, thereby realizing accurate and fast adaptive recognition of stitching actions, and effectively solving the problem of low accuracy of stitching action recognition.

[0032] The details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0034] Figure 1 is a hardware structure block diagram of a terminal according to an action recognition method of an embodiment of the present application;

[0035] Figure 2 is a flow chart of a method for motion recognition according to an embodiment of the present application;

[0036] Figure 3 is a flow chart of another action recognition method according to an embodiment of the present application;

[0037] Figure 4 is a structural schematic diagram of a target action recognition model according to an embodiment of the present application;

[0038] Figure 5 is a flow chart of another action recognition method according to an embodiment of the present application;

[0039] Figure 6 is a structural block diagram of a motion recognition device according to an embodiment of the present application;

[0040] Figure 7 It is a structural block diagram of another motion recognition device according to an embodiment of the present application. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application. In addition, it can also be understood that although the efforts made in this development process may be complex and lengthy, for ordinary technicians in the field related to the contents disclosed in the present application, some changes such as design, manufacturing or production based on the technical contents disclosed in the present application are only conventional technical means, and should not be understood as insufficient contents disclosed in the present application.

[0042] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0043] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantitative limitation, and may represent the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to greater than or equal to two. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships, for example, "A and / or B" can represent: A exists alone, A and B exist at the same time, and B exists alone. The terms "first", "second", "third" and the like involved in the present application are merely used to distinguish similar objects and do not represent a specific ordering of the objects.

[0044] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. Taking running on a terminal as an example, Figure 1 is a hardware structure block diagram of a terminal of an action recognition method according to an embodiment of the present application. Figure 1 As shown, the terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data. Optionally, the terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is for illustration only and does not limit the structure of the above terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0045] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the action recognition method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0046] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0047] This embodiment provides a method for motion recognition. Figure 2 is a flow chart of a method for motion recognition according to an embodiment of the present application. Figure 2 As shown, the process includes the following steps:

[0048] Step S210: obtaining a video segment to be tested.

[0049] The above-mentioned video segment to be tested refers to a video segment consisting of multiple frames of images collected for the suture area during the operation using an image acquisition device such as a camera or a video recorder. Since it is difficult to accurately identify an action with a single frame, in this embodiment, the acquired video segment to be tested contains at least two frames of images. It should be understood that the video segment to be tested can be a segment directly captured from the collected surgical video stream, or it can be a video segment generated by parsing the video stream of the surgical video stream into multiple frames of image frames arranged in sequence and preprocessing the image frames.

[0050] Step S220, input the video segment to be tested into the trained target action recognition model for processing; utilize the multi-head attention mechanism network in the target action recognition model to fuse the image features of each frame in the video segment to be tested, obtain the fused features, and output the action recognition result according to the fused features.

[0051] Among them, the above-mentioned target action recognition model can be generated based on the training of a multi-classification model that introduces a multi-head attention mechanism. For example, the classification model can be designed as a four-classification model, and for the suturing action in the image, four types of needle threading action, knotting action, tightening action and other actions can be identified. The classification model can be based on a convolutional neural network (CNN) combined with a multi-head attention mechanism, or built entirely based on a Transformer structure. Specifically, the CNN layer (if a CNN+attention mechanism structure is adopted) or the encoder part of the Transformer (if a pure Transformer structure is adopted) can be used to extract features of the image or video frame. The extracted features are then input into the multi-head attention network to capture the feature dependencies between different "heads", and finally the output of the multi-head attention is mapped to the probability distribution of the four categories using a fully connected layer (Dense Layer).

[0052] It can be seen that multi-head attention allows the model to learn the dependencies between different parts of the sequence in different representation subspaces in parallel by dividing the input data into multiple "heads" (or subspaces). This mechanism greatly enhances the model's ability to capture complex relationships and improves the generalization performance of the model. Therefore, in the application of intraoperative suturing action evaluation, the multi-head attention mechanism can be used to focus on different features of the surgical suture area, such as the shape, continuity, tension, etc. of the suture line. By using multiple attention heads in parallel, the suture action can be analyzed and evaluated from different perspectives to more accurately judge the quality of the suture, which helps to improve the surgical effect and patient safety.

[0053] It should be noted that after the above-mentioned action recognition results are generated, the action recognition results can be sent to the display interface for display, so as to realize real-time feedback of the action recognition results. Among them, an intuitive and easy-to-use display interface is designed according to the application scenario; the interface may include elements such as a video window, a recognition result display area, and control buttons. The results output by the action recognition model (such as "needle threading action", "knotting action", etc.) can be displayed in real time in a designated area on the interface, and text labels, icons or animations can be used to clearly indicate the type of action currently recognized. With the continuous input of video frames, the recognition results on the display interface should also be updated in real time to reflect the latest action status. Necessary interactive functions are also added to the display interface, such as pause / resume, playback, selection of different viewing angles, etc., so that users can view and operate the recognition results as needed. If necessary, a user feedback mechanism can also be added to enable users to evaluate the accuracy of the recognition results and provide feedback to improve the model. In addition, an error handling mechanism is added during data transmission and processing to deal with network delays, data loss or model anomalies.

[0054] In the above-mentioned action recognition method, the video segment to be tested is input into the target action recognition model that introduces a multi-head attention mechanism network, and the multi-frame features are fused using the multi-head attention mechanism network; by introducing multiple attention heads, the model can have the ability to pay attention to the pictures at different times at the same time, and obtain more comprehensive and accurate information, and in the forward process, multiple attention heads are processed in parallel, so that the model can model the input image from multiple angles and better capture the relationship between multiple frames, so that the weights of the mutual correlation between frames can be reasonably allocated to achieve the purpose of better identifying fine movements, thereby realizing accurate and fast adaptive recognition of stitching movements, and effectively solving the problem of low accuracy of stitching movement recognition.

[0055] In some of the embodiments, a motion recognition method is also provided. Figure 3 is a flow chart of another action recognition method according to an embodiment of the present application. Figure 3 As shown, the process includes Figure 2 The step S220 shown further includes the following steps:

[0056] Step S310, obtaining an image to be tested; performing bright channel processing on the image to be tested to obtain a target image.

[0057] The above-mentioned image to be tested refers to the current frame image parsed from the collected surgical video stream. In this step, a bright channel processing method is provided, that is, different from extracting a state in which the image appears darker, this embodiment extracts a state in which the image appears brighter. This is mainly due to the requirement that endoscopic surgery is usually performed in an in vivo environment with limited light, and the light source mainly comes from external equipment (such as surgical lights), so the bright areas in the image often contain important surgical details. Bright channel processing aims to extract the brightest areas or features from these images to highlight the details of the surgical area and improve the clarity and visibility of the image.

[0058] Step S320, obtaining a plurality of frames of images collected before and / or after the target image; and generating a video segment to be tested based on the plurality of frames of images and the target image.

[0059] The video clip is generated based on the frame of the target image obtained after the current processing. Specifically, after the pre-processed image is obtained, the key detail features have been highlighted. In order to accurately identify the stitching action, several frame images associated with the target image can be obtained.

[0060] More specifically, the above-mentioned several frame images can be multiple frame images acquired in the acquired surgical video stream within a time period before the moment corresponding to the acquisition of the target image; or, multiple frame images acquired in the surgical video stream within a time period after the moment corresponding to the acquisition of the target image can be used as the several frame images associated with the target image; or, multiple frame images acquired in the surgical video stream within a period of time before and after the moment corresponding to the acquisition of the target image can be used as the above-mentioned several frame images.

[0061] For the above-mentioned target image, it can be used together with several related frame images as the above-mentioned video segment to be tested, that is, multiple frame images including the frame target image can be used as a video segment, and the collected frames in the segment are arranged in chronological order to ensure the continuity of the video segment; and then in the subsequent steps, the video segment is input into the target action recognition model for action recognition.

[0062] Through the above steps S310 to S320, by performing bright channel processing on the surgical image and generating a video segment to be tested containing key details, it is possible to retain and enhance important details as much as possible, which is conducive to a more comprehensive evaluation and analysis of the suturing actions during the operation and improves the accuracy of action recognition.

[0063] In some embodiments, the bright channel processing of the image to be tested to obtain the target image may further include the following steps:

[0064] Perform bright channel processing on each pixel of the image to be tested to determine the maximum pixel value in the preset color channel; generate a grayscale image based on the maximum pixel value; perform maximum value filtering on the grayscale image to obtain the target image.

[0065] The above-mentioned color channels are used to describe the color composition of the image. The color channels are different in different color spaces. For example, if the image to be tested is an image in RGB format, the corresponding preset color channels include R (red) channel, G (green) channel and B (blue) channel; if the image to be tested is a LAB image, the corresponding color channels include L (brightness) channel, A (color change from green to red) channel and B (color change from blue to yellow) channel. Specifically, taking the RGB channel as an example, one of the calculation methods of the bright channel is shown in the following formula:

[0066]

[0067] In the above formula, max() is used to represent the maximum value function. J is used to represent the pixel value of each pixel in the image to be tested, x and y represent the pixel coordinates, and c represents the channel.

[0068] Through the above embodiment, the maximum pixel value of the image to be tested in the color channel is determined to form a grayscale image, and then a maximum value filtering is performed to achieve bright channel calculation; since the pixel value of the highlight area in the image in the color channel is often large, the maximum value filtering can effectively retain the highlight area in the image. Therefore, by converting the maximum pixel value into a grayscale image and then performing the maximum value filtering, the difference between different brightness areas in the image can be increased, and the highlight parts in the image can be further emphasized, so that these areas are more prominent in the final image.

[0069] In some embodiments, the bright channel processing of the image to be tested to obtain the target image includes the following steps:

[0070] Perform bright channel processing on the image to be tested to obtain a bright channel image; perform normalization processing based on the bright channel image to obtain a weight matrix; calculate the cumulative distribution function of the image to be tested; perform weighted equalization processing on the image to be tested according to the weight matrix and the cumulative value of the pixel points to obtain a target image.

[0071] After obtaining the bright channel image of the bright channel processing, the weight matrix can be obtained by normalization processing. The calculation formula of the normalization processing is as follows:

[0072] Normalized value = (pixel value - minimum value) / (maximum value - minimum value)

[0073] In this way, each pixel in the calculated image has a value between 0 and 1. The purpose of normalization is to scale the pixel values ​​to a specific range (usually 0 to 1) so that they can be used as weights. Each element in the weight matrix represents the relative brightness of the corresponding pixel in the original bright channel image.

[0074] After that, weighted image histogram equalization is performed based on the weight matrix W. Specifically, for an input image, the grayscale histogram of the input original image to be tested is first counted. Then the cumulative distribution function (CDF) of the original image is calculated, that is, the cumulative sum of the number of pixels from gray level 0 to the current gray level. Assuming that the current pixel value is x', the formula for calculating the new pixel value new_x' is as follows:

[0075] new_x'=round(255×CDF(x')×W(x'))

[0076] In the above formula, round() is used to round the number to a given number of digits. Using this formula, the original values ​​of all pixels are replaced with new pixel values ​​to form a weighted equalized image.

[0077] Through the above-mentioned embodiment, in the preprocessing stage of the image to be tested, after the bright channel calculation, the image normalization weight matrix is ​​calculated and the weighted image histogram equalization processing is performed; by redistributing the pixel values ​​of the image, the brightness distribution of the image is made more uniform, thereby enhancing the details and edges of the image, and in subsequent processing and analysis, important features in the image can be better identified, which helps to improve the accuracy and stability of the algorithm.

[0078] In some embodiments, the multi-head attention mechanism network in the target action recognition model is used to fuse the image features of each frame in the video segment to be tested to obtain the fused features, and the following steps may also be included:

[0079] Each frame image in the video segment to be tested is input into the feature extraction network in the target action recognition model respectively, and the image features corresponding to each frame image are output; each image feature is input into the multi-head attention mechanism network in parallel to obtain the fusion feature.

[0080] Specifically, multiple frames of images are input into each feature extraction network of the target action recognition network, and the corresponding features are output through each feature extraction network; wherein, for each frame of image, the corresponding feature extraction network will output a feature map, and these feature maps contain the key information of the frame of image. Next, the feature maps of each frame of image are input into the multi-head attention mechanism network in parallel, so that the model can pay attention to information from different positions at the same time when processing sequence data. In the practical application of stitching action recognition, this helps the model capture the temporal dependency and spatial relationship between different frames. Finally, the fused features are input into the fully connected layer (also called the densely connected layer). The role of the fully connected layer is to map the abstract features extracted by the previous layer to the labeled space of the sample, that is, to output the classification result.

[0081] Through the above process, the key information in multiple frames is effectively extracted, fused, and finally used for action classification. This method, which combines the CNN feature extraction capability with the multi-head attention mechanism, can significantly improve the accuracy and robustness of action recognition.

[0082] In some embodiments, the video segment to be tested includes a target image. The target image may be an image obtained by subjecting the original image to bright channel processing, or may be an image processed by bright channel and then subjected to weighted image histogram equalization. Then, the multi-head attention mechanism network in the target action recognition model is used to fuse the image features of each frame in the video segment to be tested to obtain a fused feature, and the following steps may also be included:

[0083] Using a multi-head attention mechanism network, the image features corresponding to the target image are weightedly fused with the image features corresponding to other images to obtain fused features; the other images are images other than the target image in the video segment to be tested.

[0084] More specifically, see Figure 4 , the figure shows the target action recognition model structure of this embodiment. Among them, FE is a feature extraction network; ResB is a residual network, which is mainly used to efficiently deepen the model depth; FC is a fully connected layer. Taking 15 frames of images as an example, image 8 is the target image after at least the above-mentioned bright channel processing, image 1 to image 7 are 7 frames of images collected before image 8, and image 9 to image 15 are 7 frames of images collected after image 8. Then the multiple frames of images from image 1 to image 15 are respectively input to each feature extraction network of the classification network, and the corresponding features are output through each feature extraction network. In the forward process, the features extracted from the target image 8, image 8, are fused with the extracted features of other frame images before and after based on multiple attention heads in turn, so that the key features of the image can be effectively retained during the image fusion process. In addition, after the fusion of each layer of features, the residual network can also be used to efficiently deepen the model depth; during forward propagation, the input signal can be directly propagated to the output through the residual connection, which helps to maintain the information flow in the network, so that the deep network can learn useful features more easily. Finally, the fused features are input to the fully connected layer for processing, and the corresponding recognition results indicating the suturing action categories such as needle threading action, knotting action, tightening action or other actions are output.

[0085] Through the above embodiment, the target image after bright channel processing is subjected to feature fusion based on the multi-head attention mechanism with other images in the video segment to be tested, which helps to highlight the key information in the target image, and the multi-head attention mechanism can further integrate this information and emphasize it in the fused features, thereby helping to improve the accuracy of target action recognition, especially in complex or dynamically changing scenes.

[0086] In some embodiments, training the target action recognition model comprises the following steps:

[0087] Obtain a training image set; the training image set carries multiple category labels; input the training image set into an initial action recognition model including a multi-head attention mechanism network for training, and output the predicted category results; calculate the loss function based on the predicted category results and category labels; transfer the gradient of the loss function back to the initial action recognition model for iteration to generate a target action recognition model.

[0088] Specifically, an initial model suitable for processing image data is selected, such as a convolutional neural network (CNN) with a multi-head attention mechanism, and the above training image set is input into the initial action recognition model for training. During the training process, the model will learn how to extract features from multiple frames of images and weighted feature fusion, and establish a mapping relationship between image features and action types. Among them, the training data can be images or video frames containing different action categories collected from various sources (such as public data sets, historical images, etc.). When annotating data, multiple frames of images are also used as a video clip; the number of frames of the video clip should be the same as the number of frames of the above video segment to be tested. There are four annotation types, namely 0-other, 1-needle threading, 2-knotting, and 3-tightening. Design an initial action recognition model that includes a convolutional layer (for feature extraction), a multi-head attention mechanism network (for feature fusion and enhancement), and a fully connected layer (for classification). The preprocessed and annotated training image set is input into the initial action recognition model; the model extracts image features through the convolutional layer, then performs feature fusion and enhancement through the multi-head attention mechanism network, and finally outputs the predicted category results through the fully connected layer. The predicted category results and the corresponding analog labels are compared, and the loss function results of the two are calculated. According to the loss function results, the gradient of each parameter in the model is calculated through the back propagation algorithm to iteratively optimize the model until the iteration stopping conditions such as iteration time, number of iterations or model convergence are met, and finally a fully trained target action recognition model is generated.

[0089] Through the above embodiment, the model is trained by using a training image set carrying multiple category labels to obtain a fully trained target action recognition model, which can effectively improve the accuracy of action recognition, thereby facilitating further improving the accuracy of image processing.

[0090] The following is a detailed description with reference to specific embodiments. Figure 5 is a flow chart of another action recognition method according to an embodiment of the present application. Figure 5 As shown, the process includes the following steps:

[0091] Step S501, obtaining a surgical video stream.

[0092] Step S502, input the current frame of the surgical video stream, and perform the following special image preprocessing operations on the frame image: 1. Calculate the bright channel, that is, take the grayscale image composed of the maximum value in the RGB channel, and then perform a maximum value filter. 2. After obtaining the bright channel, calculate the image normalization weight matrix. 3. Perform weighted image histogram equalization on the input image, replace the original pixel values ​​of all pixels in the original image with new pixel values, and form a weighted equalized image.

[0093] Step S503, based on the current frame, generate a video segment to be tested. Wherein, based on the current frame, a video segment is generated. After obtaining the image preprocessed in step 1, the key detail features have been highlighted. At this time, for each input frame of the picture, we take the 7 frames before and after this picture as a video segment together with it, and send this video segment to the subsequent classification model for action recognition.

[0094] Step S504: send the video segment to be tested to the target action recognition model for classification, and output the action recognition result for the video segment to be tested. Due to the fineness of stitched action recognition, the classic classification model structure is often difficult to capture the differences between various actions. Therefore, this embodiment combines the multi-head attention mechanism to propose the following Figure 4 The model structure is shown. Assuming that the current frame output is the 8th frame, the relationship between the previous and next frames of multiple frames is used to improve the model's ability to focus on different inputs. By introducing multiple attention heads, the model can be equipped with the ability to focus on the pictures at different times at the same time, thereby obtaining more comprehensive and accurate information. Finally, the recognition result of the stitching action is output.

[0095] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0096] The present embodiment also provides a kind of action recognition device, and this device is used to realize the above-mentioned embodiment and preferred implementation mode, and what has been described is not repeated here. As used below, the term "module", "unit", "subunit" etc. can realize the combination of software and / or hardware of predetermined function. Although the device described in the following embodiment is preferably realized by software, the realization of hardware, or the combination of software and hardware is also possible and conceived.

[0097] Figure 6 is a structural block diagram of a motion recognition device according to an embodiment of the present application, such as Figure 6 As shown, the device includes: an acquisition module 61, used to acquire a video segment to be tested; a recognition module 62, used to input the video segment to be tested into a trained target action recognition model for processing; using the multi-head attention mechanism network in the target action recognition model, the image features of each frame in the video segment to be tested are fused to obtain a fused feature, and the action recognition result is output based on the fused feature.

[0098] In some of the embodiments, the acquisition module 61 is also used to acquire the image to be tested; perform bright channel processing on the image to be tested to obtain a target image; the acquisition module 61 is also used to acquire several frames of images collected before and / or after the target image; and generate a video segment to be tested based on the several frames of images and the target image.

[0099] In some embodiments, the acquisition module 61 is also used to perform bright channel processing on each pixel of the image to be tested to determine the maximum pixel value in the preset color channel; the acquisition module 61 is also used to generate a grayscale image based on the maximum pixel value; the acquisition module performs maximum value filtering on the grayscale image to obtain the target image.

[0100] In some embodiments, the acquisition module 61 is also used to perform bright channel processing on the image to be tested to obtain a bright channel image; perform normalization processing based on the bright channel image to obtain a weight matrix; the acquisition module 61 is also used to calculate the cumulative distribution function of the image to be tested; and perform weighted equalization processing on the image to be tested according to the weight matrix and the cumulative distribution function to obtain a target image.

[0101] In some of the embodiments, the above-mentioned recognition module 62 is also used to input each frame image in the video segment to be tested into the feature extraction network in the target action recognition model, and output the image features corresponding to each frame image; the recognition module 62 is also used to input each image feature in parallel into the multi-head attention mechanism network to obtain a fusion feature.

[0102] In some of the embodiments, the video segment to be tested includes a target image; the recognition module is also used to use a multi-head attention mechanism network to weightedly fuse image features corresponding to the target image with image features corresponding to other images to obtain fused features; the other images are images other than the target image in the video segment to be tested.

[0103] Figure 7 is a structural block diagram of another motion recognition device according to an embodiment of the present application, such as Figure 7 As shown, the device comprises Figure 6 All modules shown in the figure also include: a training module 71 for obtaining a training image set; the training image set carries multiple category labels; the training module 71 is also used to input the training image set into an initial action recognition model including a multi-head attention mechanism network for training, and output a predicted category result; the loss function is calculated based on the predicted category result and the category label; the training module 71 is also used to reversely transfer the gradient of the loss function to the initial action recognition model for iteration to generate a target action recognition model.

[0104] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0105] This embodiment further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0106] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0107] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0108] S1, obtaining a video segment to be tested.

[0109] S2, input the video segment to be tested into the trained target action recognition model for processing; use the multi-head attention mechanism network in the target action recognition model to fuse the image features of each frame in the video segment to be tested, obtain the fused features, and output the action recognition result based on the fused features.

[0110] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0111] In addition, in combination with the action recognition method in the above embodiment, the embodiment of the present application can provide a storage medium for implementation. The storage medium stores a computer program; when the computer program is executed by a processor, any one of the action recognition methods in the above embodiment is implemented.

[0112] Those skilled in the art should understand that the technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0113] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for motion recognition, characterized in that: The method comprises: Get the video segment to be tested; The video segment to be tested is input into the trained target action recognition model for processing; the multi-head attention mechanism network in the target action recognition model is used to fuse the image features of each frame in the video segment to be tested to obtain a fused feature, and the action recognition result is output according to the fused feature.

2. The action recognition method according to claim 1, characterized in that: The method of using the multi-head attention mechanism network in the target action recognition model to fuse the image features of each frame in the video segment to be tested to obtain the fused features includes: Inputting each frame image in the video segment to be tested into the feature extraction network in the target action recognition model, and outputting the image features corresponding to each frame image; Input each of the image features in parallel into the multi-head attention mechanism network to obtain the fused features.

3. The action recognition method according to claim 1, characterized in that: The step of obtaining the video segment to be tested includes: Acquire an image to be tested; perform bright channel processing on the image to be tested to obtain a target image; Acquire a plurality of frames of images collected before and / or after the target image; and generate the video segment to be tested based on the plurality of frames of images and the target image.

4. The action recognition method according to claim 3, characterized in that: The step of performing bright channel processing on the image to be tested to obtain a target image includes: Performing bright channel processing on each pixel of the image to be tested to determine the maximum pixel value in a preset color channel; and generating a grayscale image based on the maximum pixel value; Perform maximum value filtering on the grayscale image to obtain the target image.

5. The action recognition method according to claim 3, characterized in that: The step of performing bright channel processing on the image to be tested to obtain a target image includes: Performing bright channel processing on the image to be tested to obtain a bright channel image; performing normalization processing based on the bright channel image to obtain a weight matrix; The cumulative distribution function of the image to be tested is calculated; and according to the weight matrix and the cumulative distribution function, the image to be tested is subjected to weighted equalization processing to obtain the target image.

6. The motion recognition method according to any one of claims 3 to 5, characterized in that: The method of using the multi-head attention mechanism network in the target action recognition model to fuse the image features of each frame in the video segment to be tested to obtain the fused features includes: The multi-head attention mechanism network is used to weightedly fuse the image features corresponding to the target image with the image features corresponding to other images to obtain the fused features; the other images are images other than the target image in the video segment to be tested.

7. The action recognition method according to claim 1, characterized in that: Training the target action recognition model includes: Obtaining a training image set; the training image set carries multiple category labels; Inputting the training image set into an initial action recognition model including the multi-head attention mechanism network for training, and outputting a predicted category result; A loss function is calculated according to the predicted category result and the category label; and the gradient of the loss function is back-transferred to the initial action recognition model for iteration to generate the target action recognition model.

8. A motion recognition device, characterized in that: The device comprises: An acquisition module, used to acquire the video segment to be tested; The recognition module is used to input the video segment to be tested into the trained target action recognition model for processing; utilize the multi-head attention mechanism network in the target action recognition model to fuse the image features of each frame in the video segment to be tested, obtain the fusion feature, and output the action recognition result according to the fusion feature.

9. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to execute the action recognition method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute the motion recognition method according to any one of claims 1 to 7 when running.

Images