A firing frame recognition method, device and equipment

Abstract

The present application relates to the field of medical image processing technology, and in particular to a method, device and equipment for identifying a firing frame. The method includes: in the process of receiving image frames transmitted sequentially by an image acquisition device in a time sequence, the received N consecutive image frames are determined as input data; wherein N≥2; the input data is input into a feature extraction module to obtain N image features; the N image features are input into a self-attention module to obtain N attention vectors; the N attention vectors are input into a feature dimension reduction module to obtain the predicted probability of each image frame being a firing frame; the predicted probability that meets the preset standard among the N predicted probabilities is determined as the firing frame corresponding to the image frame; the present application can solve the technical problem of low determination efficiency in the existing firing frame determination method.

Description

Technical Field

[0001] The present application relates to the field of medical image processing technology, and in particular to a firing frame recognition method, device and equipment. Background Art

[0002] In the medical field, a biopsy needle can be used to puncture a patient's organ tissue site for sampling; in certain operational scenarios, in order to obtain sufficient samples from a target site in the patient's body, it may be necessary to perform sampling operations at multiple different sampling locations in the target site.

[0003] In actual applications, multiple sampling operations and prolonged sampling operations may increase patient discomfort. Therefore, when sampling operations are required at multiple sampling locations, the sampling locations where sampling operations have been performed should be accurately recorded while performing the sampling operations quickly and accurately to prevent repeated sampling at the same sampling location.

[0004] In the prior art, the doctor can record the moment when he presses the switch of the biopsy needle, which is the firing moment of the sampling operation. The firing moment is the moment when the sampling is successful. Subsequently, based on the firing moment, the firing frame with the shooting moment as the firing moment can be determined from multiple ultrasound image frames recording the sampling operation. The image content of the firing frame can indicate the sampling position of the sampling operation.

[0005] In actual operation, whether manually recording the firing moment or determining the firing frame in multiple ultrasound image frames, the time required to determine the firing frame will be greatly prolonged. Therefore, it can be seen that the existing firing frame determination method has the technical problem of low determination efficiency. Summary of the invention

[0006] In view of this, the purpose of the present application is to provide a firing frame identification method, device and equipment to solve the technical problem of low determination efficiency in the existing firing frame determination method.

[0007] In a first aspect, the present application provides a firing frame recognition method, which is applied to a firing frame recognition device, wherein the firing frame recognition device is configured with a firing frame recognition model, and the model includes a feature extraction module, a self-attention module, and a feature dimension reduction module; the firing frame recognition device is connected to an image acquisition device; the method includes:

[0008] In the process of receiving the image frames transmitted sequentially by the image acquisition device in a time sequence, the received N consecutive image frames are determined as input data; wherein N≥2;

[0009] Inputting the input data into the feature extraction module to obtain N image features;

[0010] Inputting the N image features into the self-attention module to obtain N attention vectors;

[0011] Inputting N attention vectors into the feature dimension reduction module to obtain the predicted probability that each image frame is a firing frame;

[0012] The image frame corresponding to the prediction probability that meets the preset standard among the N prediction probabilities is determined as the firing frame.

[0013] In a second aspect, the present application provides a firing frame recognition device, which is applied to a firing frame recognition device, wherein the firing frame recognition device is configured with a firing frame recognition model, wherein the model includes a feature extraction module, a self-attention module, and a feature dimension reduction module; the firing frame recognition device is connected to an image acquisition device; the device includes: an image receiving module and a firing frame recognition module;

[0014] The image receiving module is used to determine the received N consecutive image frames as input data during the process of receiving the image frames transmitted sequentially by the image acquisition device in a time sequence; wherein N≥2;

[0015] The firing frame recognition module is used to input the input data into the feature extraction module to obtain N image features;

[0016] The firing frame recognition module is used to input the N image features into the self-attention module to obtain N attention vectors;

[0017] The firing frame recognition module is used to input the N attention vectors into the feature dimension reduction module to obtain the predicted probability that each of the image frames is a firing frame;

[0018] The firing frame identification module is used to determine the image frame corresponding to the prediction probability that meets the preset standard among the N prediction probabilities as the firing frame.

[0019] In a third aspect, the present application provides a firing frame recognition device, which includes a processor and a memory, wherein the memory is used to store an application program, and the processor runs or executes a software program stored in the memory so that the firing frame recognition device implements the above-mentioned firing frame recognition method.

[0020] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store program codes executed by a processor, wherein the program codes are used to implement the above-mentioned firing frame recognition method.

[0021] In a fifth aspect, the present application provides a computer program product, which includes computer instructions. When the computer instructions are executed on an electronic device, the electronic device implements the above-mentioned firing frame recognition method.

[0022] Beneficial effects:

[0023] The present application provides a trigger frame recognition method, the method comprising: in the process of receiving image frames transmitted sequentially by an image acquisition device in a time sequence, determining the received N continuous image frames as input data; inputting the input data into a feature extraction module to obtain N image features; inputting the N image features into a self-attention module to obtain N attention vectors; inputting the N attention vectors into a feature dimension reduction module to obtain a prediction probability that each image frame is a trigger frame; determining the image frame corresponding to the prediction probability that meets a preset standard among the N prediction probabilities as a trigger frame;

[0024] In summary, the firing frame recognition method provided in the present application can apply the firing frame recognition model to perform real-time processing on N image frames. The self-attention module in the firing frame recognition model can capture the global relationship between the image features of the N image frames, which is used to better capture the temporal change characteristics before and after the needle tip is fired, improve the recognition accuracy of the firing frames in a fast motion state, and reduce the dependence on the quality of a single image frame; the feature dimensionality reduction module in the firing frame recognition model can reduce the dimension and classify the attention vector. Compared with the use of the fully connected layer for classification in the prior art, the feature dimensionality reduction module in the present application can improve the processing speed, thereby meeting the real-time processing of firing frame recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. The following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0026] Figure 1 A flowchart of a firing frame recognition method provided in an embodiment of the present application;

[0027] Figure 2 A schematic diagram of the structure of a firing frame recognition system provided in an embodiment of the present application;

[0028] Figure 3 A schematic diagram of the firing frame recognition model provided in an embodiment of the present application for identifying input data;

[0029] Figure 4A schematic diagram of the relationship between input data and target image frames provided in an embodiment of the present application;

[0030] FIG5( a ) is an example diagram of input data provided in an embodiment of the present application;

[0031] FIG5( b ) is another example diagram of input data provided in an embodiment of the present application;

[0032] Figure 6 A schematic diagram of the structure of a firing frame recognition device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0033] In the existing method for identifying the firing frame, the doctor mainly manually records the firing moment so as to determine the firing frame in a plurality of ultrasound image frames recording the sampling operation. However, this method has the technical problem of low efficiency.

[0034] In actual operation, a sensor can also be installed on the biopsy needle to record the firing moment through an electronic sensor, but installing the sensor on the biopsy needle will increase the mass of the biopsy needle, etc.; in addition, if the installation position of the sensor is offset during the use of the biopsy needle, etc., the sensor will interfere with the stability of the biopsy needle, etc., and thus affect the running trajectory of the biopsy needle, etc. Therefore, the method of installing the sensor on the biopsy needle will affect the accuracy of the sampling operation, so this method cannot be promoted.

[0035] In summary, the existing firing frame determination method has the technical problem of low determination efficiency.

[0036] In order to solve the above technical problems, the present application provides a technical solution for trigger frame recognition, which is applied to a trigger frame recognition device 100, in which a trigger frame recognition model is deployed, and the trigger frame recognition model can recognize the trigger frame in N image frames; the trigger frame recognition device 100 is connected to an image acquisition device 200 and a visualization device 300; in actual operation, ① the image acquisition device 200 is used to acquire N image frames of a sampling operation and transmit the N image frames to the trigger frame recognition device 100 in sequence; ② the trigger frame recognition device 100 recognizes the N image frames of input data through the trigger frame recognition model to obtain the predicted probability of each image frame, and then determines the trigger frame in the N image frames according to the predicted probability, and the trigger frame recognition device 100 sends the trigger frame to the visualization device 300; ③ the visualization device 300 visualizes the trigger frame.

[0037] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

[0038] First, the present application provides a firing frame recognition method, which is applied to a firing frame recognition device 100 in a firing frame recognition system. The firing frame recognition device 100 is configured with a firing frame recognition model, and the model includes a feature extraction module, a self-attention module, and a feature dimension reduction module; the firing frame recognition system also includes: an image acquisition device 200 and a visualization device 300; the firing frame recognition device 100 is connected to the image acquisition device 200 and the visualization device 300; Figure 1 and Figure 2 As shown, Figure 1 A flowchart of a firing frame recognition method provided in an embodiment of the present application is shown in FIG. Figure 2 This is a schematic diagram of the structure of the firing frame recognition system provided in an embodiment of the present application. The method includes: S110 to S150, and the details are as follows:

[0039] S110: In the process of receiving image frames transmitted sequentially by an image acquisition device according to a time sequence, the received N consecutive image frames are determined as input data; wherein N≥2.

[0040] Specifically, in the embodiment of the present application, the image acquisition device 200 may be a device specifically used to acquire image frames related to sampling operations, or it may be a medical device with an image acquisition function; after the image acquisition device 200 acquires N image frames in sequence, the N image frames are transmitted in sequence to the trigger frame recognition device 100, so that the trigger frame recognition device 100 recognizes the trigger frame among the N image frames in real time.

[0041] The firing frame recognition model provided in the embodiment of the present application can determine the movement trend of the sampling part (such as the needle tip in the biopsy needle) in the biopsy needle shown in the image content based on the image content of multiple consecutive image frames, so as to determine the firing frame in multiple consecutive image frames.

[0042] Input data N image frames Input data In actual operation, after the trigger frame recognition device 100 receives N image frames, it performs standardization processing on the N image frames. The standardization processing process includes: adjusting the size of the image frames and normalizing the pixel values ​​of the image frames. Figure 3 As shown, Figure 3A schematic diagram of the processing of input data by the firing frame recognition model provided in an embodiment of the present application. After standardization processing is performed on each of N image frames, N 1*224*224 image frames are obtained, where N is the number of image frames that are continuous in time sequence and input to the firing frame recognition model, and N is a positive integer greater than 1.

[0043] It should be emphasized that the firing frame recognition method provided in the embodiment of the application needs to be executed for each of the multiple image frames corresponding to each sampling operation. If the firing frame recognition method provided in the embodiment of the application is implemented to identify the firing frame among the multiple image frames corresponding to the current sampling operation, it is necessary to continue to implement the firing frame recognition method provided in the embodiment of the application to identify the firing frame among the multiple image frames corresponding to the next sampling operation, and so on.

[0044] In actual operation, the firing frame recognition device 100 needs to perform "marking processing" on different "multiple image frames recording sampling operations" to distinguish different "multiple image frames recording sampling operations" and prevent the firing frame corresponding to the current sampling operation from being mistakenly confirmed as the firing frame corresponding to the next sampling operation.

[0045] S120: Input the input data to a feature extraction module to obtain N image features.

[0046] Specifically, the firing frame recognition device 100 inputs N image frames into the feature extraction module, and enables the feature extraction module to extract features of each image frame to obtain image features of each image frame.

[0047] In an embodiment of the present application, the feature extraction module includes N groups of stacked convolutional neural networks (CNN), each group of stacked convolutional neural networks includes a plurality of convolutional neural networks of sequential vectors; each group of stacked convolutional neural networks performs image feature extraction on an image frame to obtain image features corresponding to each image frame; in actual operation, the network parameters of the N groups of stacked convolutional networks are the same.

[0048] like Figure 3 As shown, after the image features of N image frames are extracted by the feature extraction module, N 1024*7*7 feature maps are obtained, and the feature maps are image features.

[0049] S130: Input N image features into the self-attention module to obtain N attention vectors.

[0050] Specifically, in the embodiment of the present application, the self-attention module is a set of stacked Transformer Encoders, that is, Figure 3In the encoder, when N image frames are input into the self-attention module together, the self-attention module can capture the global relationship of the image features of the N image frames, which is used to better capture the temporal change characteristics before and after the needle tip is fired, improve the recognition accuracy of the firing frames in a fast motion state, and reduce the dependence on the quality of a single image frame.

[0051] In one implementation, S130 includes: Step (1) to Step (3), the details of which are as follows:

[0052] Step (1): Determine the position code of each image frame according to the arrangement order of each image frame in the N consecutive image frames included in the input data.

[0053] Specifically, unlike the one-to-one input between N image frames and N groups of stacked convolutional neural networks, when the image features of N image frames are input into the self-attention module, the image features of the N image frames are input into the self-attention module at the same time. Therefore, it is necessary to determine the position encoding of each image frame. The position encoding is used to indicate the arrangement order of the image frame in the input data. The position encoding is used to establish a one-to-one correspondence between the N attention vectors and the image features of the N image frames when the self-attention module outputs N attention vectors.

[0054] Step (2): For each image frame, the image features and position codes corresponding to the image frame are spliced ​​to obtain a first splicing vector.

[0055] Specifically, in actual operation, before splicing the image features with the position codes, global average pooling is required to compress the image features into 1*1024 feature vectors. After obtaining the position codes of each image frame, the 1*1024 feature vectors and position codes corresponding to each image frame are spliced ​​to obtain the first splicing vectors corresponding to each image frame.

[0056] Step (3): Input the N first splicing vectors into the self-attention module, determine the attention weight of each first splicing vector through the self-attention module, and multiply each first splicing vector with the corresponding attention weight to obtain N attention vectors.

[0057] The attention weight indicates the degree of correlation between a first splicing vector and the remaining first splicing vectors.

[0058] Specifically, Figure 3 As shown, after N first splicing vectors are input into the self-attention module, the self-attention module determines the attention weight of each first splicing vector, and multiplies each first splicing vector with the corresponding attention weight to obtain N 1*1024 attention vectors.

[0059] S140: Input the N attention vectors into the feature dimension reduction module to obtain the predicted probability that each image frame is a firing frame.

[0060] Specifically, in the embodiment of the present application, the feature dimension reduction module has both the function of feature dimension reduction and the function of classification. When multiple attention vectors are input into the feature dimension reduction module, the predicted probability of each image frame being a firing frame can be determined; in practical applications, the feature dimension reduction module includes N groups of stacked one-dimensional convolutional neural networks.

[0061] In one implementation, the multiple attention vectors correspond to the N image frames in a one-to-one manner; S140 includes: steps (4) to (6), the details of which are as follows:

[0062] Step (4): Concatenate the N attention vectors to obtain a second concatenated vector.

[0063] Specifically, Figure 3 As shown, after the self-attention module outputs N 1*1024 attention vectors, before the N attention vectors are input into the feature dimensionality reduction module, the N attention vectors need to be concatenated to obtain a second concatenated vector of 1*1024*4.

[0064] Step (5): input the second concatenated vector into a feature dimension reduction module, and perform feature dimension reduction processing on the second concatenated vector through the feature dimension reduction module to obtain a low-dimensional vector matrix.

[0065] Specifically, after the second splicing vector is obtained, the second splicing vector is input into the feature dimension reduction module, and the feature dimension reduction module performs feature dimension reduction processing on the second splicing vector to obtain a 1*1*4 low-dimensional vector matrix.

[0066] Step (6): Determine the prediction probability corresponding to each of the N image frames according to the low-dimensional vector matrix.

[0067] Specifically, after the low-dimensional vector matrix is ​​obtained, it is processed through N sigmoid activation functions to obtain the prediction probability corresponding to each image frame.

[0068] In actual operation, since the feature dimensionality reduction module that performs the classification function in the embodiment of the present application also has the function of feature dimensionality reduction, the efficiency of the output prediction probability can be greatly improved, and the time requirement for the firing frame in the sampling operation can be met to the greatest extent, so that the doctor can have sufficient time to master the firing frame corresponding to the previous sampling operation, and further master the sampling position corresponding to the previous sampling operation, thereby improving the sampling efficiency and sampling accuracy of the overall sampling operation.

[0069] In one implementation, S110 includes: step (7), details of which are as follows:

[0070] Step (7): in the process of receiving N image frames transmitted sequentially by the image acquisition device in a time sequence, each time a current image frame is received, the current image frame and (N-1) consecutive historical image frames preceding the current image frame are determined as input data;

[0071] The current image frame is an image frame received at the current moment, and the historical image frame is an image frame received at a historical moment before the current moment.

[0072] Specifically, in the embodiment of the present application, since the input data of the firing frame recognition model can only include N image frames, that is, the firing frame recognition model can only recognize N image frames at a time, but in actual operation, the number of "multiple image frames recording each sampling operation" is usually much larger than N, and therefore it is necessary to extract multiple input data in the "multiple image frames recording each sampling operation" in turn, and then input the multiple input data one by one into the firing frame recognition model for recognition.

[0073] In the embodiment of the present application, whenever a current image frame is received, the current image frame and the (N-1) consecutive historical image frames before the current image frame are determined as input data, which is equivalent to selecting multiple input data in sequence from the "multiple image frames recording each sampling operation" in a sliding window manner; in the embodiment of the present application, the sliding step length is 1 image frame, such as Figure 4 As shown, Figure 4 The process of determining multiple input data provided in the embodiment of the present application includes: Figure 4 In the t-6 " represents an image frame in "a plurality of image frames recording each sampling operation", and a dotted line box marked with "the (T-3)th input data" represents an input data in a plurality of input data sequentially determined by means of a sliding window; Figure 4 It can be seen that, in the process of determining the input data in the embodiment of the present application, multiple recognition processes may be performed on some image frames, such as the image frame marked as “P t-3 " Therefore, the embodiment of the present application can ensure the recognition accuracy of the firing frame.

[0074] In addition, after S140, the method further includes: step (8) to step (10), the details of which are as follows:

[0075] Step (8): for N input data that are continuous in time sequence, determine N prediction probabilities corresponding to the target image frame commonly included in the N input data.

[0076] Specifically, Figure 4 As shown, Figure 4 The four input data shown have a label "P t-3 " image frame, which is marked as "P t-3 The image frame of " is Figure 4 The four input data shown in the figure all include a target image frame. Since the target image frame belongs to the four input data, four firing frame recognition processes (ie, S120 to S140) are performed, and therefore it has four prediction probabilities; wherein the values ​​of the four prediction probabilities may be the same or different.

[0077] In actual operation, not all image frames have N prediction probabilities, because the multiple image frames at the beginning and the multiple image frames at the end of the "multiple image frames recording each sampling operation" are usually not firing frames. Therefore, the number of prediction probabilities corresponding to each image frame in the "multiple image frames at the beginning" and "multiple image frames at the end" mentioned above will generally be less than N, so the firing frame recognition model will directly determine the above image frames as non-firing frames; assuming Figure 4 The last image frame P shown in t is an image frame in “multiple image frames located at the end”, then the number of prediction probabilities of the image frame is less than N.

[0078] Step (9): Determine the weighted prediction probability of the target image frame according to the N prediction probabilities of the target image frame.

[0079] Specifically, after the N prediction probabilities of the target image frame are obtained, the weighted prediction probability of the target image frame can be determined according to the N probabilities of the target image frame.

[0080] It should be emphasized that when the number of predicted probabilities of the target image frames is less than N, the weighted predicted probability of the target image frame can be determined based on the predicted probability of the actual number of target image frames. If the number of predicted probabilities of the image frames is less than the preset number, it indicates that the image frame is one of the “multiple image frames located at the beginning” or the “multiple image frames located at the end”. Since the “multiple image frames located at the beginning” or the “multiple image frames located at the end” are usually directly determined as non-trigger frames by the trigger frame recognition model, there is no need to determine the weighted prediction probability for the above-mentioned image frames.

[0081] In one implementation, step (9) includes: step (9.1) to step (9.2), the details of which are as follows:

[0082] Step (9.1): Determine a weight for each of the N prediction probabilities of the target image frame based on a normal distribution.

[0083] Specifically, in the embodiment of the present application, for the N predicted probabilities corresponding to the T-th input data, N weight points are determined and mapped to the [-2, 2] interval of the standard normal distribution. There is a one-to-one correspondence between the N weight points and the N predicted probabilities. The formula for determining the N weight points is as follows:

[0084] x [k] = -2 + 4(k-1) / (n-1);

[0085] In the formula, x [k] Represents the weight corresponding to the kth prediction probability in the Nth prediction probability; k = 1,2,...,N.

[0086] After determining N weight points, determine the weight corresponding to each weight point. The formula for determining the weight is as follows:

[0087] w [k] = exp(-x [k] ² / 2);

[0088] In the formula, w [k] Represents the weight corresponding to the kth prediction probability in the Nth prediction probability.

[0089] After the weights are determined, they are normalized. The normalization formula is as follows:

[0090] W [k] = w [k] / Σw [k] ;

[0091] Where W [k] The normalized weight corresponding to the kth prediction probability in the Nth prediction probability.

[0092] Step (9.2): Perform weighted averaging of the N weights and the N predicted probabilities to obtain the weighted predicted probability.

[0093] Specifically, the formula for determining the weighted prediction probability is as follows:

[0094] P = Σ(W [k] * p [k] );

[0095] Where P represents the weighted prediction probability of the target image frame, p [k] represents the k-th predicted probability of the target image frame.

[0096] S150: Determine the image frame corresponding to the prediction probability that meets the preset standard among the N prediction probabilities as the firing frame.

[0097] Specifically, in an embodiment of the present application, an image frame that meets a preset standard is determined as a firing frame. The preset standard may be a preset threshold, that is, when the value of the predicted probability is greater than the preset threshold, such as 0.9, the corresponding image frame is determined as a firing frame.

[0098] After determining the firing frame, the firing frame recognition device 100 simultaneously sends the firing frame and the corresponding timestamp to the visualization device 300 , and the visualization device 300 displays the firing frame and the needle track of the sampling operation.

[0099] As shown in Figures 5(a) and 5(b), Figure 5(a) is an example diagram of input data provided in an embodiment of the present application, and Figure 5(b) is another example diagram of input data provided in an embodiment of the present application. The input data shown in Figures 5(a) and 5(b) are input data including 3 image frames, and the image frames marked with "OK" in the above input data are firing frames, and the image frames marked with "NG" are non-firing frames.

[0100] In one implementation, S150 includes: step (10), details of which are as follows:

[0101] Step (10): According to the weighted prediction probability of each image frame in the N consecutive image frames included in the input data, the image frame corresponding to the weighted prediction probability that meets the preset standard is determined as the firing frame.

[0102] Specifically, for an image frame with a weighted prediction probability calculated, the weighted prediction probability of the image frame is compared with a preset threshold. If the weighted prediction probability is greater than the preset threshold, the image frame corresponding to the weighted prediction probability is determined as a firing frame.

[0103] In the second aspect, the present application provides a firing frame recognition device, which is applied to a firing frame recognition device. The firing frame recognition device is configured with a firing frame recognition model, which includes a feature extraction module, a self-attention module and a feature dimension reduction module; the firing frame recognition device is connected to an image acquisition device; Figure 6 As shown, Figure 6 This is a schematic diagram of the structure of a firing frame recognition device provided in an embodiment of the present application, the device includes: an image receiving module 310 and a firing frame recognition module 320;

[0104] The image receiving module 310 is used to determine the received N consecutive image frames as input data in the process of receiving the image frames transmitted sequentially by the image acquisition device in a time sequence; wherein N≥2;

[0105] The firing frame recognition module 320 inputs the input data into the feature extraction module to obtain N image features; the firing frame recognition module 320 is also used to input the N image features into the self-attention module to obtain N attention vectors;

[0106] The firing frame identification module 320 is further used to input the N attention vectors into the feature dimension reduction module to obtain the predicted probability that each image frame is a firing frame;

[0107] The firing frame identification module 320 is further used to determine the image frame corresponding to the prediction probability that meets the preset standard among the N prediction probabilities as the firing frame.

[0108] In one implementation, the firing frame identification module 320 is further used to determine the position code of each image frame according to the arrangement order of each image frame in the N consecutive image frames included in the input data;

[0109] The firing frame identification module 320 is further used to perform splicing processing on the image features and position codes corresponding to each image frame to obtain a first splicing vector;

[0110] The firing frame identification module 320 is further configured to input the N first splicing vectors into the self-attention module, determine the attention weight of each first splicing vector through the self-attention module, and multiply each first splicing vector by the corresponding attention weight to obtain N attention vectors;

[0111] The attention weight indicates the degree of correlation between a first splicing vector and the remaining first splicing vectors.

[0112] In one implementation, the N attention vectors correspond to the N image frames in a one-to-one correspondence; the trigger frame identification module 320 is further used to splice the N attention vectors to obtain a second splicing vector;

[0113] The firing frame identification module 320 is further used to input the second splicing vector into the feature dimension reduction module, and perform feature dimension reduction processing on the second splicing vector through the feature dimension reduction module to obtain a low-dimensional vector matrix;

[0114] The firing frame identification module 320 is further used to determine the prediction probability corresponding to each image frame in the N image frames according to the low-dimensional vector matrix.

[0115] In one implementation, the image receiving module 310 is further configured to, in a process of receiving N image frames sequentially transmitted by the image acquisition device in a time sequence, determine the current image frame and (N-1) consecutive historical image frames preceding the current image frame as input data whenever a current image frame is received;

[0116] The trigger frame identification module 320 is further used to, in the process of receiving N image frames transmitted sequentially by the image acquisition device in a time sequence, determine the current image frame and (N-1) consecutive historical image frames before the current image frame as input data each time a current image frame is received; wherein the current image frame is an image frame received at the current moment, and the historical image frame is an image frame received at a historical moment before the current moment;

[0117] The firing frame identification module 320 is further used to determine, for N input data that are continuous in time sequence, N prediction probabilities corresponding to the target image frame commonly included in the N input data;

[0118] The firing frame identification module 320 is further used to determine the weighted prediction probability of the target image frame according to the N prediction probabilities of the target image frame.

[0119] In one implementation, the firing frame identification module 320 is further configured to determine a weight for each of the N predicted probabilities of the target image frame based on a normal distribution;

[0120] The firing frame identification module 320 is further used to perform weighted averaging processing on the N weights and the N predicted probabilities to obtain a weighted predicted probability.

[0121] In one implementation, the firing frame identification module 320 is further used to determine, based on the weighted prediction probability of each image frame in N consecutive image frames included in the input data, an image frame corresponding to a weighted prediction probability that meets a preset standard as a firing frame.

[0122] Third, the present application also provides a firing frame recognition device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, steps S110 to S150 provided in the above embodiment are implemented.

[0123] Fourth, the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, steps S110 to S150 of the above embodiment are executed.

[0124] Fifth, the computer program product provided in the present application includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the method in the previous method embodiment. The specific implementation can refer to steps S110~S150 of the method embodiment, which will not be repeated here.

[0125] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0126] In addition, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0127] Furthermore, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0128] It should be noted that if the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0129] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0130] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A firing frame recognition method, characterized in that: Applied to a firing frame recognition device, wherein the firing frame recognition device is configured with a firing frame recognition model, the model comprising a feature extraction module, a self-attention module and a feature dimension reduction module; The firing frame recognition device is connected to the image acquisition device; the method comprises: In the process of receiving the image frames transmitted sequentially by the image acquisition device in a time sequence, the received N consecutive image frames are determined as input data; wherein N≥2; Inputting the input data into the feature extraction module to obtain N image features; Inputting the N image features into the self-attention module to obtain N attention vectors; Inputting N attention vectors into the feature dimension reduction module to obtain the predicted probability that each image frame is a firing frame; Determine the image frame corresponding to the prediction probability that meets the preset standard among the N prediction probabilities as the firing frame; The firing frame is an ultrasonic image frame captured at the moment of successful sampling among a plurality of ultrasonic image frames recording the sampling operation; There is a one-to-one correspondence between the N attention vectors and the N image frames; the N attention vectors are input into the feature dimension reduction module to obtain the predicted probability of each image frame being a firing frame, and further includes: Concatenating the N attention vectors to obtain a second concatenated vector; Inputting the second splicing vector into the feature dimension reduction module, and performing feature dimension reduction processing on the second splicing vector by the feature dimension reduction module to obtain a low-dimensional vector matrix; Determining the prediction probability corresponding to each of the N image frames according to the low-dimensional vector matrix; In the process of receiving the image frames transmitted sequentially by the image acquisition device according to the time sequence, determining the received N consecutive image frames as input data includes: In the process of receiving N image frames transmitted sequentially by the image acquisition device in a time sequence, each time a current image frame is received, the current image frame and (N-1) consecutive historical image frames preceding the current image frame are determined as the input data; The current image frame is the image frame received at the current moment, and the historical image frame is the image frame received at a historical moment before the current moment; After inputting the N attention vectors into the feature dimension reduction module to obtain the predicted probability that each image frame is a firing frame, the method further includes: For N input data that are continuous in time sequence, determine N prediction probabilities corresponding to the target image frame commonly included in the N input data; Determining a weighted prediction probability of the target image frame according to the N prediction probabilities of the target image frame; Determining the weighted prediction probability of the target image frame according to the N prediction probabilities of the target image frame includes: Determine a weight for each of the N predicted probabilities of the target image frame based on a normal distribution; The weighted predicted probability is obtained by performing weighted averaging processing on the N weights and the N predicted probabilities.

2. The method according to claim 1, characterized in that The step of inputting the input data into the feature extraction module to obtain N image features includes: Determine the position code of each of the image frames according to the arrangement order of each of the image frames in the N consecutive image frames included in the input data; For each of the image frames, the image features and the position codes corresponding to the image frames are spliced ​​to obtain a first splicing vector; Inputting the N first splicing vectors into the self-attention module, determining the attention weight of each first splicing vector through the self-attention module, and multiplying each first splicing vector by the corresponding attention weight to obtain the N attention vectors; The attention weight indicates a correlation degree between one of the first splicing vectors and the remaining first splicing vectors.

3. The method according to claim 1, characterized in that The step of determining the image frame corresponding to the prediction probability that meets a preset standard among the N prediction probabilities as the firing frame includes: According to the weighted prediction probability of each of the N consecutive image frames included in the input data, the image frame corresponding to the weighted prediction probability that meets a preset standard is determined as the firing frame.

4. A firing frame recognition device, characterized in that: The method for implementing claim 1 is applied to a firing frame recognition device, wherein the firing frame recognition device is provided with a firing frame recognition model, wherein the model includes a feature extraction module, a self-attention module, and a feature dimension reduction module; The firing frame recognition device is connected to the image acquisition device; the device comprises: an image receiving module and a firing frame recognition module; The image receiving module is used to determine the received N consecutive image frames as input data in the process of receiving the image frames transmitted sequentially by the image acquisition device in a time sequence; wherein N≥2; The firing frame recognition module is used to input the input data into the feature extraction module to obtain N image features; The firing frame recognition module is used to input the N image features into the self-attention module to obtain N attention vectors; The firing frame recognition module is used to input the N attention vectors into the feature dimension reduction module to obtain the predicted probability that each of the image frames is a firing frame; The firing frame identification module is used to determine the image frame corresponding to the prediction probability that meets the preset standard among the N prediction probabilities as the firing frame.

5. A firing frame recognition device, characterized in that: The firing frame recognition device includes a processor and a memory, the memory is used to store an application program, and the processor runs or executes a software program stored in the memory so that the firing frame recognition device implements the firing frame recognition method as described in any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program codes executed by a processor, and the program codes are used to implement the firing frame recognition method according to any one of claims 1 to 3.

7. A computer program product, characterized in that The computer program product includes computer instructions. When the computer instructions are executed on an electronic device, the electronic device implements the firing frame recognition method as described in any one of claims 1 to 3.

Images