Pointer identification method, device and equipment of pointer type instrument, medium and product
By obtaining the spatiotemporal feature fusion of the current frame and adjacent frame images, and using bidirectional attention and convolutional gated recurrent units to process the pointer recognition of pointer instruments, the problem of low pointer recognition accuracy in rainy and foggy weather is solved, and higher recognition accuracy and real-time performance are achieved.
Patent Information
- Application Number
- CN202510880765.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-27
- Publication Date
- 2025-09-26
AI Technical Summary
In complex weather conditions such as rain and fog, the traditional method of identifying the pointer of a pointer-type instrument after removing the rain and fog has low accuracy or may even be impossible to identify.
By obtaining the current frame image and adjacent frame images, the bidirectional attention mechanism and convolutional gated recurrent unit are used to extract spatiotemporal features, and the spatial and channel attention mechanisms are combined to perform feature fusion, which is then input into the pre-trained pointer recognition model for recognition.
It improves the accuracy and reliability of pointer recognition, simplifies the process, enhances the anti-interference ability to complex weather, and improves real-time performance.
Smart Images
Figure CN120708203A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of instrument detection, and in particular to a pointer identification method, apparatus, computer equipment, computer-readable storage medium, and computer program product for a pointer-type instrument. Background Art
[0002] In order to solve the problem of how to identify the pointer of pointer instruments in complex weather conditions such as rain and fog, traditional technology adopts a strategy based on removing rain and fog before identification. However, when faced with the influence of more severe rain and fog, this strategy is not effective in removing rain and fog, resulting in reduced accuracy of pointer identification, and even failure to identify the pointer position.
[0003] Therefore, there is an urgent need for a method that can accurately identify the pointer of a pointer-type instrument in rainy and foggy weather. Summary of the Invention
[0004] Based on this, it is necessary to provide a pointer recognition method, device, computer equipment, computer-readable storage medium and computer program product for a pointer instrument that can improve the pointer recognition accuracy of a pointer instrument in order to address the above technical problems.
[0005] In a first aspect, the present application provides a pointer identification method for a pointer-type instrument, comprising:
[0006] Obtaining the current frame image and adjacent frame images of the pointer instrument to be identified;
[0007] Obtaining spatiotemporal features of the current frame image according to the current frame image and the adjacent frame images;
[0008] Obtaining, based on the spatiotemporal features of the current frame image and the current frame image, spatiotemporal feature weights of the adjacent frame images relative to the current frame image;
[0009] Acquiring the spatiotemporal features of the adjacent frame images, and obtaining fused image features corresponding to the current frame image based on the spatiotemporal features of the adjacent frame images, the spatiotemporal features of the current frame image, and the spatiotemporal feature weights;
[0010] The fused image features are input into a pre-trained pointer recognition model to obtain a pointer recognition result of the pointer-type instrument to be recognized in the current frame image.
[0011] In one embodiment, obtaining the spatiotemporal features of the current frame image according to the current frame image and the adjacent frame images includes:
[0012] Acquire current frame visual features corresponding to the current frame image and adjacent frame visual features corresponding to the adjacent frame images;
[0013] Using a bidirectional attention mechanism, obtaining a current frame attention feature corresponding to the current frame visual feature and an adjacent frame attention feature corresponding to the adjacent frame visual feature;
[0014] Extracting the temporal relationship between the current frame attention feature and the adjacent frame attention feature through a convolutional gated recurrent unit;
[0015] According to the temporal relationship and the attention features of the current frame, the spatiotemporal features of the current frame image are obtained.
[0016] In one embodiment, the bidirectional attention mechanism includes a spatial attention mechanism and a channel attention mechanism;
[0017] The method of using a bidirectional attention mechanism to obtain a current frame attention feature corresponding to the current frame visual feature includes:
[0018] Utilizing a spatial attention mechanism, obtaining a spatial attention feature corresponding to the visual feature of the current frame;
[0019] Utilizing the channel attention mechanism, obtaining the channel attention feature corresponding to the visual feature of the current frame;
[0020] Based on the product result of the spatial attention feature and the channel attention feature and the spatial attention feature, the current frame attention feature corresponding to the current frame visual feature is obtained.
[0021] In one embodiment, obtaining the spatiotemporal feature weights of the adjacent frame images relative to the current frame image based on the spatiotemporal features of the current frame image and the current frame image includes:
[0022] Performing a splicing operation on the spatiotemporal features of the current frame image and the current frame visual features corresponding to the current frame image in a channel dimension to obtain a splicing feature;
[0023] The splicing features are subjected to a stacked convolution transformation operation to obtain spatiotemporal feature weights of the adjacent frame images relative to the current frame image.
[0024] In one embodiment, obtaining the fused image features corresponding to the current frame image according to the spatiotemporal features of the adjacent frame images, the spatiotemporal features of the current frame image, and the spatiotemporal feature weights includes:
[0025] The spatiotemporal features of the adjacent frame images and the spatiotemporal features of the current frame image are weighted and summed according to the spatiotemporal feature weights to obtain a fused image feature corresponding to the current frame image.
[0026] In one embodiment, the training process of the pointer recognition model includes:
[0027] Acquiring a visual image of a training pointer instrument and marking the actual pointer position in the visual image;
[0028] Acquire a current frame sample image and adjacent frame sample images of the training pointer instrument from the visual image;
[0029] Obtaining spatiotemporal features of the current frame sample image according to the current frame sample image and the adjacent frame sample images;
[0030] Obtaining, according to the spatiotemporal features of the current frame sample image and the current frame sample image, spatiotemporal feature weights of the adjacent frame sample images relative to the current frame sample image;
[0031] Acquiring spatiotemporal features of adjacent frame sample images, and obtaining fused image features corresponding to the current frame sample image based on the spatiotemporal features of the adjacent frame sample images, the spatiotemporal features of the current frame sample image, and the spatiotemporal feature weights of the adjacent frame sample images relative to the current frame sample image;
[0032] Inputting the fused image features corresponding to the current frame sample image into the pointer recognition model to be trained, and obtaining a pointer position prediction result of the pointer instrument to be identified in the current frame sample image;
[0033] A loss value is obtained according to a difference between the pointer position prediction result and the actual pointer position marked in the current frame sample image, and the pointer recognition model is trained using the loss value.
[0034] In a second aspect, the present application further provides a pointer identification device for a pointer-type instrument, comprising:
[0035] An image acquisition module is used to acquire a current frame image and adjacent frame images of the pointer instrument to be identified;
[0036] a feature fusion module, configured to obtain, based on the current frame image and the adjacent frame images, the spatiotemporal features of the current frame image; obtain, based on the spatiotemporal features of the current frame image and the current frame image, the spatiotemporal feature weights of the adjacent frame images relative to the current frame image; obtain the spatiotemporal features of the adjacent frame images, and obtain, based on the spatiotemporal features of the adjacent frame images, the spatiotemporal features of the current frame image, and the spatiotemporal feature weights, the fused image features corresponding to the current frame image;
[0037] The pointer recognition module is used to input the fused image features into a pre-trained pointer recognition model to obtain a pointer recognition result of the pointer-type instrument to be identified in the current frame image.
[0038] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method described in the first aspect when executing the computer program.
[0039] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the method described in the first aspect when the computer program is executed by a processor.
[0040] In a fifth aspect, the present application further provides a computer program product, comprising a computer program, which implements the steps of the method described in the first aspect when executed by a processor.
[0041] The aforementioned pointer recognition method, apparatus, computer device, computer-readable storage medium, and computer program product for a pointer-type instrument obtain the spatiotemporal features of the current frame image by acquiring the current frame image and adjacent frame images. Based on the spatiotemporal features of the current frame image and the current frame image, the spatiotemporal feature weights of the adjacent frame images relative to the current frame image are then determined. Based on the spatiotemporal feature weights, the spatiotemporal features of the adjacent frame images and the current frame image are then weightedly fused to obtain fused graphical features. The fused graphical features are then input into a pre-trained pointer recognition model to obtain pointer recognition results. This feature fusion step fully utilizes the spatiotemporal information of the current frame image and adjacent frame images, enhancing the current frame image features and improving the ability to resist interference from complex weather conditions (such as rain and fog), thereby increasing the accuracy and reliability of pointer recognition. Furthermore, complex image enhancement operations such as rain and fog removal are not required, simplifying the process and improving the real-time performance of pointer recognition. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments of the present application or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying any creative work.
[0043] Figure 1 A diagram illustrating an application environment of a pointer recognition method for a pointer-type instrument according to an embodiment;
[0044] Figure 2 Schematic diagram of a flow chart of a pointer identification method for a pointer-type instrument in one embodiment;
[0045] Figure 3 Schematic diagram of a process for obtaining spatiotemporal features of a current frame image in one embodiment;
[0046] Figure 4 A diagram of a local application environment of a method for identifying pointer-type instruments in complex weather conditions based on feature fusion in one embodiment;
[0047] Figure 5 Schematic diagram of the structure of a channel attention module in one embodiment;
[0048] Figure 6 Schematic diagram of the structure of a feature fusion module in one embodiment;
[0049] Figure 7 A structural block diagram of a pointer identification device for a pointer instrument in one embodiment;
[0050] Figure 8 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION
[0051] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.
[0052] The pointer identification method of the pointer instrument provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The server 104 terminal 102 obtains the current frame image and adjacent frame images of the pointer instrument to be identified; the server 104 obtains the spatiotemporal features of the current frame image based on the current frame image and the adjacent frame images; the server 104 obtains the spatiotemporal feature weights of the adjacent frame images relative to the current frame image based on the spatiotemporal features of the current frame image and the current frame image; the server 104 obtains the spatiotemporal features of the adjacent frame images, and obtains the fused image features corresponding to the current frame image based on the spatiotemporal features of the adjacent frame images, the spatiotemporal features of the current frame image, and the spatiotemporal feature weights; the server 104 inputs the fused image features into a pre-trained pointer recognition model to obtain the pointer recognition result of the pointer instrument to be identified in the current frame image. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, inspection robots, tablet computers and other devices. The server 104 may be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.
[0053] In an exemplary embodiment, Figure 2 As shown, a pointer identification method for a pointer instrument is provided, and the method is applied to Figure 1 The server 104 in the example is used as an example to illustrate the process, including the following steps S202 to S210.
[0054] Step S202 : obtaining a current frame image and adjacent frame images of the pointer instrument to be identified.
[0055] The pointer-type instrument to be identified may refer to various outdoor pointer-type instruments whose pointer positions and readings need to be identified, such as pressure gauges in industrial equipment.
[0056] The current frame image may refer to a single image of the pointer instrument to be identified collected at the current moment, and may be a single image of the instrument containing rain and fog interference.
[0057] Among them, the adjacent frame image can refer to the image that is adjacent to the current frame image in the time series, which can provide the previous and next correlation information and assist in the feature analysis of the current frame image. The adjacent frame image can be one frame image or multiple frame images, which can be set according to actual needs.
[0058] Exemplarily, the server may obtain the current frame image and adjacent frame images of the pointer instrument to be identified through a terminal, wherein the terminal may be a monitoring terminal of the pointer instrument on site, used to photograph the pointer instrument to be identified through an image acquisition device.
[0059] Step S204 : obtaining the spatiotemporal features of the current frame image according to the current frame image and adjacent frame images.
[0060] Among them, spatiotemporal features can refer to abstract feature representations that fuse spatial information (pointer shape and position in the image, etc.) with temporal information (the change relationship between the current frame and adjacent frames, such as the pointer change relationship).
[0061] Exemplarily, the server extracts visual features from the current frame image and the adjacent frame images respectively, then analyzes the feature relationship between the visual features of the current frame and the adjacent frames, and then obtains the spatiotemporal features of the current frame image.
[0062] Step S206 : Obtain the spatiotemporal feature weights of the adjacent frame images relative to the current frame image based on the spatiotemporal features of the current frame image and the current frame image.
[0063] The spatiotemporal feature weight may refer to a coefficient matrix that quantifies the contribution of adjacent frames to the feature enhancement of the current frame.
[0064] Exemplarily, the server calculates the spatiotemporal feature weights of adjacent frame images relative to the current frame image based on the spatiotemporal features of the current frame image and the visual features of the current frame image.
[0065] Step S208 , obtaining the spatiotemporal features of adjacent frame images, and obtaining the fused image features corresponding to the current frame image according to the spatiotemporal features of the adjacent frame images, the spatiotemporal features of the current frame image, and the spatiotemporal feature weights.
[0066] Among them, fusing image features can refer to fusing the spatiotemporal features of adjacent frame images with the spatiotemporal features of the current frame image according to the spatiotemporal feature weights, thereby enhancing the features of the current frame image and obtaining the feature representation of the current frame image after feature enhancement.
[0067] Exemplarily, the server can obtain the spatiotemporal features of adjacent frame images based on the adjacent frame images and the previous frame images of the adjacent frames, and then perform weighted fusion on the spatiotemporal features of the adjacent frame images and the spatiotemporal features of the current frame image based on the obtained spatiotemporal feature weights to obtain the fused image features corresponding to the current frame image.
[0068] In step S210 , the fused image features are input into a pre-trained pointer recognition model to obtain a pointer recognition result of the pointer-type instrument to be recognized in the current frame image.
[0069] The pointer recognition model may be a model that has been trained to accurately detect the position of a pointer in a pointer-type instrument based on input fused image features. The pointer recognition result may include the position information of the pointer.
[0070] Exemplarily, the server inputs the fused image features into a pre-trained pointer recognition model, and the pointer recognition model outputs a pointer recognition result of the pointer-type instrument to be recognized in the current frame image through internal classification and regression operations.
[0071] In the aforementioned pointer recognition method for a pointer-type instrument, the spatiotemporal features of the current frame image are obtained by acquiring the current frame image and adjacent frame images. Based on the spatiotemporal features of the current frame image and the current frame image, the spatiotemporal feature weights of the adjacent frame images relative to the current frame image are then determined. A weighted fusion of the spatiotemporal features of the adjacent frame images and the current frame image is then performed based on the spatiotemporal feature weights to obtain a fused graphical feature. This fused graphical feature is then input into a pre-trained pointer recognition model to obtain a pointer recognition result. This feature fusion step fully utilizes the spatiotemporal information of the current frame image and adjacent frame images, enhancing the current frame image features and improving the ability to resist interference from complex weather conditions (such as rain and fog), thereby increasing the accuracy and reliability of pointer recognition. Furthermore, complex image enhancement operations such as rain and fog removal are not required, simplifying the process and improving the real-time performance of pointer recognition.
[0072] In an exemplary embodiment, Figure 3 As shown, obtaining the spatiotemporal features of the current frame image based on the current frame image and adjacent frame images includes the following steps:
[0073] Step S302 , obtaining current frame visual features corresponding to the current frame image and adjacent frame visual features corresponding to adjacent frame images.
[0074] The current frame visual features may refer to features extracted from the current frame image that reflect the visual content of the image, such as color, texture, etc. The adjacent frame visual features may refer to features extracted from adjacent frame images that reflect their visual content.
[0075] Exemplarily, the server may utilize a preset feature extraction technology to extract corresponding current frame visual features and adjacent frame visual features from the current frame image and the adjacent frame image, respectively.
[0076] In step S304, a bidirectional attention mechanism is used to obtain the current frame attention features corresponding to the current frame visual features and the adjacent frame attention features corresponding to the adjacent frame visual features.
[0077] The bidirectional attention mechanism can refer to a mechanism that can focus on and enhance features from both spatial and channel dimensions, highlighting important spatial regions and key channel features. The current frame attention feature can refer to a more representative and important feature obtained after the visual features of the current frame are processed by the bidirectional attention mechanism. The adjacent frame attention feature can refer to a more representative and important feature obtained after the visual features of adjacent frames are processed by the bidirectional attention mechanism.
[0078] Exemplarily, the server may use a bidirectional attention mechanism to process the current frame visual features and the adjacent frame visual features respectively, to obtain the current frame attention features corresponding to the current frame visual features and the adjacent frame attention features corresponding to the adjacent frame visual features.
[0079] Step S306: extract the temporal relationship between the attention features of the current frame and the attention features of adjacent frames through a convolutional gated recurrent unit.
[0080] Among them, the Convolutional Gated Recurrent Unit (ConvGRU) can refer to a neural network model that combines convolution operations and gated recurrent unit structures. It can be used to capture the timing information between the current frame and adjacent frames, and learn their changing patterns and associations in the time dimension by processing the attention features of the current frame and adjacent frames.
[0081] Among them, the temporal relationship may refer to the correlation and change relationship between the visual features of the current frame and the attention features of the adjacent frames in the time series.
[0082] Step S308: Obtain the spatiotemporal features of the current frame image based on the temporal relationship and the attention features of the current frame.
[0083] For example, the convolutional gated recurrent unit adopts a recursive convolutional unit structure, which can process spatial and temporal features at the same time, capture spatial features through convolution operations, and use the gating mechanism to control the flow of information in the time dimension, thereby updating the hidden state. The server first inputs the attention features of the adjacent frames into the convolutional gated recurrent unit to update the hidden state, and then the server inputs the attention features of the current frame into the convolutional gated recurrent unit to update the hidden state again. The hidden state after the update is the temporal relationship. The convolutional gated recurrent unit uses the updated hidden state and the current frame attention features to comprehensively generate the spatiotemporal features of the current frame image. In this process, the convolutional gated recurrent unit learns the temporal relationship between the attention features of the current frame and the attention features of the adjacent frames, such as information such as the direction and speed of movement of the pointer between the adjacent frames and the current frame.
[0084] In this embodiment, the visual features of the current frame and adjacent frames are processed separately through a bidirectional attention mechanism, and the temporal relationship between them is mined using a convolutional gated recurrent unit. The bidirectional attention mechanism enables the server to focus on the spatial regions and channel information in the image that are critical for pointer recognition, preventing the server from being disturbed by irrelevant information, thereby more accurately extracting features valuable for pointer recognition. The convolutional gated recurrent unit further captures the temporal relationship between the features of the current frame and adjacent frames, and can learn the dynamic changes of the pointer. For example, in complex weather conditions, the pointer may be partially blurred due to interference such as raindrops and fog, but by analyzing adjacent frames, the model can use the temporal relationship to infer the exact position of the pointer. Compared with considering only the features of a single frame, the accuracy of feature mining is improved, thereby improving the accuracy of pointer recognition.
[0085] In an exemplary embodiment, the bidirectional attention mechanism includes a spatial attention mechanism and a channel attention mechanism; using the bidirectional attention mechanism to obtain the current frame attention feature corresponding to the current frame visual feature can include the following steps: using the spatial attention mechanism to obtain the spatial attention feature corresponding to the current frame visual feature; using the channel attention mechanism to obtain the channel attention feature corresponding to the current frame visual feature; based on the product result of the spatial attention feature and the channel attention feature and the spatial attention feature, obtain the current frame attention feature corresponding to the current frame visual feature.
[0086] The spatial attention mechanism can refer to a mechanism that focuses on the spatial position information of an image, and is used to highlight important spatial regions by calculating the contribution of each spatial position to the final feature map. The channel attention mechanism can refer to a mechanism that focuses on the channel information of an image, and by analyzing the feature map of each channel, determines the contribution of different channels to the final feature map, and highlights key channel features.
[0087] Exemplarily, the server uses the spatial attention mechanism to obtain the spatial attention features corresponding to the visual features of the current frame; the server uses the channel attention mechanism to obtain the channel attention features corresponding to the visual features of the current frame; the server obtains the product result of the spatial attention features and the channel attention features, and then combines the product result with the spatial attention features to obtain the current frame attention features corresponding to the visual features of the current frame.
[0088] In this embodiment, a spatial attention mechanism focuses on the spatial position of the image, while a channel attention mechanism focuses on channel information, generating spatial attention features and channel attention features, respectively. The spatial attention mechanism can focus on specific spatial regions in the image related to the pointer, avoiding interference from other irrelevant areas. For example, in complex weather conditions, there may be interference factors such as rain and fog, but the spatial attention mechanism can focus on the dial area where the pointer is located. The channel attention mechanism highlights channel information closely related to pointer features, such as the channel corresponding to pointer color, further enhancing the ability to capture pointer features. By combining the spatial and channel attention mechanisms, the feature specificity is improved, allowing for greater focus on information useful for pointer recognition, thereby improving the accuracy of pointer recognition in complex weather conditions. By combining the product operation with the spatial attention features, important information from the spatial and channel dimensions is deeply integrated, enriching the feature expression. When faced with image quality degradation caused by complex weather conditions, this fused feature can provide more comprehensive and representative information, thereby improving the accuracy of subsequent pointer recognition.
[0089] In an exemplary embodiment, based on the spatiotemporal features of the current frame image and the current frame image, the spatiotemporal feature weights of the adjacent frame images relative to the current frame image are obtained, which can specifically include the following steps: splicing the spatiotemporal features of the current frame image and the current frame visual features corresponding to the current frame image in the channel dimension to obtain spliced features; performing a stacked convolution transformation operation on the spliced features to obtain the spatiotemporal feature weights of the adjacent frame images relative to the current frame image.
[0090] The splicing operation may refer to splicing the spatiotemporal features of the current frame image and the visual features of the current frame corresponding to the current frame image according to the channel dimension, thereby combining the feature information of the two. The stacked convolution transformation operation may refer to performing multiple convolution operations on the spliced features and performing transformation operations such as activation functions.
[0091] Exemplarily, the server can perform a splicing operation on the spatiotemporal features of the current frame image and the current frame visual features corresponding to the current frame image in the channel dimension to integrate the time information and spatial information contained in the spatiotemporal features and the intuitive visual information contained in the current frame visual features to obtain spliced features; the server then performs multiple convolution operations on the spliced features, and through conversion operations such as activation functions, obtains the spatiotemporal feature weights of adjacent frame images relative to the current frame image.
[0092] In this embodiment, the feature information of two different dimensions, spatiotemporal and visual, is integrated together through a splicing operation, providing the model with more comprehensive feature data. The stacked convolution transformation operation deeply explores the potential relationship between these different dimensional features through multi-level feature extraction and transformation. For example, the pointer movement trend information in the spatiotemporal features is combined with the pointer color and shape information in the visual features of the current frame. The model can learn the connection between the change in pointer appearance and the change in position in the time series, thereby providing a richer and more accurate basis for determining the spatiotemporal feature weights of adjacent frames relative to the current frame. This deep fusion and relationship mining can better capture the inherent laws of image features and improve the rationality and accuracy of weight calculation.
[0093] In an exemplary embodiment, obtaining the fused image features corresponding to the current frame image based on the spatiotemporal features of adjacent frame images, the spatiotemporal features of the current frame image, and the spatiotemporal feature weights may include the steps of performing weighted summation of the spatiotemporal features of the adjacent frame images and the spatiotemporal features of the current frame image based on the spatiotemporal feature weights to obtain the fused image features corresponding to the current frame image.
[0094] The weighted summation may refer to performing weighted calculation on each corresponding element in the spatiotemporal features of adjacent frame images and the spatiotemporal features of the current frame image.
[0095] Exemplarily, the server performs weighted summation of the spatiotemporal features of adjacent frame images and the spatiotemporal features of the current frame image according to the spatiotemporal feature weights to obtain the fused image features corresponding to the current frame image.
[0096] In this embodiment, feature fusion is achieved by weighting the spatiotemporal features of adjacent frames and the current frame using spatiotemporal feature weights. This weighted summation effectively integrates spatiotemporal information from different moments. This fusion provides more comprehensive pointer motion and position information, enhancing the expressiveness of the features and making the fused image features more representative and discriminative, thus facilitating more accurate pointer identification in complex environments.
[0097] In an exemplary embodiment, the training process of the pointer recognition model includes: obtaining a visual image of a training pointer instrument and marking the actual pointer position in the visual image; obtaining a current frame sample image and an adjacent frame sample image of the training pointer instrument from the visual image; obtaining the spatiotemporal features of the current frame sample image based on the current frame sample image and the adjacent frame sample image; obtaining the spatiotemporal feature weights of the adjacent frame sample image relative to the current frame sample image based on the spatiotemporal features of the current frame sample image and the current frame sample image; obtaining the spatiotemporal features of the adjacent frame sample image, and obtaining a fused image feature corresponding to the current frame sample image based on the spatiotemporal features of the adjacent frame sample image, the spatiotemporal features of the current frame sample image, and the spatiotemporal feature weights of the adjacent frame sample image relative to the current frame sample image; inputting the fused image feature corresponding to the current frame sample image into the pointer recognition model to be trained to obtain a pointer position prediction result of the pointer instrument to be identified in the current frame sample image; obtaining a loss value based on the difference between the pointer position prediction result and the actual pointer position marked in the current frame sample image, and using the loss value to train the pointer recognition model.
[0098] Among them, the training pointer instrument may refer to a pointer instrument specifically used to train the pointer recognition model, and its visual image data is used to help the model learn the characteristics of the pointer under different conditions to improve the model's accuracy in pointer recognition. The actual pointer position may refer to the actual position of the pointer in the visual image of the training pointer instrument, which is accurately marked by humans. The current frame sample image may refer to an image sample representing the current moment selected from the visual image sequence of the training pointer instrument. The adjacent frame sample image may refer to an image sample that is adjacent to the current frame sample image in the time series.
[0099] For example, a server can use image acquisition equipment to capture a large amount of visual image data from a training pointer instrument in different weather conditions (e.g., rain, fog, and sunshine), at different angles, and under different lighting conditions. The server can then manually annotate each visual image to precisely indicate the actual pointer position. From these annotated visual images, an image is then selected in chronological order as the current frame sample image, and one or more frames adjacent to it in the temporal sequence are simultaneously selected as adjacent frame sample images. This selection can be done at fixed intervals, for example, selecting adjacent frames every five frames, or the selection strategy can be dynamically adjusted based on actual conditions to ensure the model learns temporal variation characteristics to varying degrees. The server then obtains the spatiotemporal features of the current frame sample image based on the current frame sample image and the adjacent frame sample images; obtains the spatiotemporal feature weights of the adjacent frame sample images relative to the current frame sample image based on the spatiotemporal features of the current frame sample image and the current frame sample image; obtains the spatiotemporal features of the adjacent frame sample images, and obtains the fused image features corresponding to the current frame sample image based on the spatiotemporal features of the adjacent frame sample images, the spatiotemporal features of the current frame sample image, and the spatiotemporal feature weights of the adjacent frame sample images relative to the current frame sample image; the server inputs the fused image features corresponding to the current frame sample image into the pointer recognition model to be trained, and obtains the pointer position prediction result of the pointer instrument to be identified in the current frame sample image; the server obtains a loss value based on the difference between the pointer position prediction result and the actual pointer position marked in the current frame sample image, and uses the loss value to train the pointer recognition model through the back propagation algorithm.
[0100] In this embodiment, by precisely annotating the actual pointer position in the visual image of the training pointer instrument, the precise annotation of the actual pointer position provides accurate supervision information for model training, enabling the model to clearly define its learning objectives. Spatiotemporal feature fusion effectively integrates the spatiotemporal information of adjacent frames and the current frame, enhancing the expressive power of the input features and providing the model with richer and more discriminative information. Backpropagation and parameter optimization based on loss values enable the model to continuously adjust its parameters to reduce the difference between the predicted results and the actual position. The above technical means work together to enable the model to continuously learn and improve during the training process, thereby improving the accuracy of pointer position recognition.
[0101] In an exemplary embodiment, the present application also provides a method for identifying pointer instruments in complex weather conditions based on feature fusion, which is applied to Figure 4 、 Figure 5 as well as Figure 6 In the application environment shown, the method includes:
[0102] Step S11 : obtaining a current frame image and adjacent frame images of the pointer instrument to be identified.
[0103] The server may first obtain the current frame image and adjacent frame images of the pointer instrument to be identified from the terminal.
[0104] Step S12, obtaining the current frame visual features corresponding to the current frame image and the adjacent frame visual features corresponding to the adjacent frame images; the bidirectional attention mechanism includes a spatial attention mechanism and a channel attention mechanism; utilizing the spatial attention mechanism, obtaining the spatial attention features corresponding to the current frame visual features; utilizing the channel attention mechanism, obtaining the channel attention features corresponding to the current frame visual features; based on the product result of the spatial attention features and the channel attention features and the spatial attention features, obtaining the current frame attention features corresponding to the current frame visual features; extracting the temporal relationship between the current frame attention features and the adjacent frame attention features through a convolutional gated recurrent unit; obtaining the spatiotemporal features of the current frame image based on the temporal relationship and the current frame attention features.
[0105] For example, the current frame visual features corresponding to the current frame image and the adjacent frame visual features corresponding to the adjacent frame images can be obtained by the feature extraction module. For example, the visual image data corresponding to the current frame image Input into the feature extraction module, the visual features Fv (current frame visual features) corresponding to the current frame image can be extracted, where C is the number of channels, H and W are the height and width of the image respectively; the feature extraction module can be obtained based on the backbone network of YOLOv5 (You Only Look Once version 5). Figure 4 As shown, the visual features of the current frame are input as input features, and the spatial attention module 401 calculates the attention weight map Fq. The size of Fq is the same as the visual features of the current frame. The weight value of each position represents the contribution of the position to the final feature map. Finally, the spatial attention module performs a tensor multiplication operation on Fq and the visual features of the current frame to obtain the weighted spatial attention features corresponding to the visual features of the current frame. Figure 5 As shown, the channel attention module 402 first performs a pooling operation on the feature map of each channel to obtain a C-dimensional vector X. The weight of each position in the vector represents the contribution of different channels to the final feature map. Then, through three fully connected layers and 1x1 convolution kernels, 3x3 convolution kernels, and 1x1 convolution kernels, and a sigmoid function, the current frame visual feature is multiplied by the X vector in the number of channels to obtain the channel attention feature corresponding to the current frame visual feature; then, through the following Figure 4 shown (element-level multiplication operation) obtains the product of the spatial attention feature and the channel attention feature, and then (Sum operation) adds the product result and the spatial attention feature to obtain the current frame attention feature corresponding to the current frame visual feature; in order to reduce the number of channels of the current frame attention feature, the current frame attention feature can be input into a 1x1 convolution kernel for convolution operation for compression; the compressed current frame attention feature is then input into the convolution gated recurrent unit 403. The convolution gated recurrent unit 403 adopts a recursive convolution unit, which can process spatial and temporal features at the same time. The compressed current frame attention feature is fused with the features in the convolution gated recurrent unit at the past moment (features corresponding to adjacent frames) by using the convolution operation and the gating mechanism, and the hidden state is updated. After processing through the ReLU activation function, the spatiotemporal features (output features) of the current frame image containing spatiotemporal information (temporal relationship) are obtained.
[0106] Step S13: Perform a splicing operation on the channel dimension on the spatiotemporal features of the current frame image and the current frame visual features corresponding to the current frame image to obtain a spliced feature; perform a stacked convolution transformation operation on the spliced feature to obtain the spatiotemporal feature weights of the adjacent frame images relative to the current frame image. The spatiotemporal features of the adjacent frame images are obtained, and a weighted sum of the spatiotemporal features of the adjacent frame images and the spatiotemporal features of the current frame image is performed based on the spatiotemporal feature weights to obtain a fused image feature corresponding to the current frame image.
[0107] like Figure 6 As shown, the external memory stores spatiotemporal features of multiple time steps, from left to right, the spatiotemporal features of the current time step t (the spatiotemporal features of the current frame image), the spatiotemporal features of time step t-1, the spatiotemporal features of time step t-2, and the spatiotemporal features of time step t-3. In this embodiment, the spatiotemporal features of adjacent frame images can refer to the spatiotemporal features of time steps t-1 to t-3. That is, in this embodiment, adjacent frames represent the three frames from time steps t-1 to t-3. It should be understood that the spatiotemporal features of adjacent frame images can be obtained by processing the adjacent frame visual features of the adjacent frame images through the steps in step S12. The processing in step S12 should be performed in a time sequence. That is, in the process of obtaining the spatiotemporal features, the image at time step t-3 is first processed and the spatiotemporal features are obtained, followed by the image at time step t-2 and the spatiotemporal features, followed by the image at time step t-1 and the spatiotemporal features, and finally the image at the current time step t is processed and the spatiotemporal features are obtained. The server inputs the visual features of the current frame and the features in the external memory into the feature fusion module. The visual features of the current frame are resized by the 1x1 convolution layer and then input into the splicing layer. The spatiotemporal features of the current time step t in the external memory are input into the splicing layer. The splicing layer performs a splicing operation on the received features in the channel dimension and then outputs them to the stacked convolution layer. The stacked convolution layer (such as Figure 6 The two consecutive convolutional layers after the splicing layer are combined with the ReLU function for conversion operation, and then the global average pooling (GAP) and Softmax function are used to calculate the temporal soft attention weights. The temporal soft attention weights are then weighted with the four spatiotemporal features from time step t to time step t-3 input from the external memory through tensor product, and then fused through element-by-element summation. The number of channels of the feature map is then adjusted through a 1x1 convolution layer to finally obtain an enhanced fused feature image (the fused image features corresponding to the current frame image).
[0108] Step S14: input the fused image features into a pre-trained pointer recognition model to obtain a pointer recognition result of the pointer-type instrument to be recognized in the current frame image.
[0109] The pointer recognition model training steps include: collecting visual image data of outdoor pointer instruments in different weather conditions and annotating the pointer positions; ensuring that the dataset contains different types of image data from different scenarios to ensure the model's generalization capabilities. Steps S12 and S13 are then performed on the annotated dataset to obtain the corresponding fused image features. These fused image features are then input into the pointer recognition model to be trained, predicting the pointer's position information. A loss function is used to calculate the error between the model output and the annotated pointer position. Backpropagation is then used to update the pointer recognition model's parameter weights to optimize the model's detection performance. These steps are repeated until the model converges, resulting in a trained pointer recognition model. The trained pointer recognition model can accurately detect the pointer position of pointer instruments in complex weather conditions.
[0110] In this embodiment, the feature extraction module is used to extract low-level features from the image; the spatial attention module and the channel attention module are used to enhance feature extraction capabilities, allowing the model to focus on important areas and features in the image; the convolutional gated recurrent unit is used to extract the temporal information of the image and obtain the spatiotemporal features of the current frame; the feature fusion module calculates the fusion weight based on the spatiotemporal features, and performs a weighted fusion of the features of the current frame with the features of the past frames to obtain the enhanced fused image features of the current frame, thereby improving the accuracy of pointer recognition. Through feature fusion, the features of the current frame image are enhanced, the model's ability to resist interference in complex weather conditions is improved, and the accuracy and reliability of pointer recognition are improved. In addition, in this embodiment, there is no need to perform image enhancement operations such as rain and fog removal, which simplifies the process and improves the real-time performance and efficiency of pointer recognition.
[0111] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily performed in sequence in the order indicated by the arrows. Unless clearly stated herein, the execution of these steps is not strictly limited in order, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of the steps or stages in other steps or other steps. It is understandable that the various steps in different embodiments can be freely combined as needed, and the various non-contradictory schemes formed by the combination all fall within the scope of protection of this application.
[0112] Based on the same inventive concept, embodiments of the present application also provide a pointer identification device for a pointer-type instrument, which is used to implement the aforementioned pointer identification method for a pointer-type instrument. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more embodiments of the pointer identification device for a pointer-type instrument provided below can be found in the aforementioned limitations of the pointer identification method for a pointer-type instrument, and will not be further elaborated here.
[0113] In an exemplary embodiment, Figure 7 As shown, a pointer recognition device 900 for a pointer instrument is provided, comprising: an image acquisition module 901, a feature fusion module 902 and a pointer recognition module 903, wherein:
[0114] The image acquisition module 901 is used to acquire the current frame image and adjacent frame images of the pointer instrument to be identified;
[0115] Feature fusion module 902 is configured to obtain the spatiotemporal features of the current frame image based on the current frame image and the adjacent frame images; obtain the spatiotemporal feature weights of the adjacent frame images relative to the current frame image based on the spatiotemporal features of the current frame image and the current frame image; obtain the spatiotemporal features of the adjacent frame images, and obtain the fused image features corresponding to the current frame image based on the spatiotemporal features of the adjacent frame images, the spatiotemporal features of the current frame image, and the spatiotemporal feature weights;
[0116] The pointer recognition module 903 is used to input the fused image features into a pre-trained pointer recognition model to obtain a pointer recognition result of the pointer-type instrument to be recognized in the current frame image.
[0117] In an exemplary embodiment, the feature fusion module 902 is also used to obtain the current frame visual features corresponding to the current frame image and the adjacent frame visual features corresponding to the adjacent frame image; utilize the bidirectional attention mechanism to obtain the current frame attention features corresponding to the current frame visual features and the adjacent frame attention features corresponding to the adjacent frame visual features; extract the temporal relationship between the current frame attention features and the adjacent frame attention features through the convolutional gated recurrent unit; and obtain the spatiotemporal features of the current frame image based on the temporal relationship and the current frame attention features.
[0118] In an exemplary embodiment, the bidirectional attention mechanism includes a spatial attention mechanism and a channel attention mechanism; the above-mentioned feature fusion module 902 is also used to use the spatial attention mechanism to obtain the spatial attention features corresponding to the current frame visual features; use the channel attention mechanism to obtain the channel attention features corresponding to the current frame visual features; based on the product result of the spatial attention features and the channel attention features and the spatial attention features, obtain the current frame attention features corresponding to the current frame visual features.
[0119] In an exemplary embodiment, the feature fusion module 902 is further used to perform a splicing operation on the spatiotemporal features of the current frame image and the current frame visual features corresponding to the current frame image in the channel dimension to obtain a spliced feature; and perform a stacked convolution transformation operation on the spliced feature to obtain the spatiotemporal feature weights of the adjacent frame images relative to the current frame image.
[0120] In an exemplary embodiment, the feature fusion module 902 is further configured to perform weighted summation of the spatiotemporal features of adjacent frame images and the spatiotemporal features of the current frame image according to the spatiotemporal feature weights to obtain fused image features corresponding to the current frame image.
[0121] In an exemplary embodiment, the pointer recognition device 900 for the pointer instrument further includes a model training module for obtaining a visual image of a pointer instrument for training and marking the actual pointer position in the visual image; obtaining a current frame sample image and an adjacent frame sample image of the pointer instrument for training from the visual image; obtaining the spatiotemporal features of the current frame sample image based on the current frame sample image and the adjacent frame sample image; obtaining the spatiotemporal feature weights of the adjacent frame sample image relative to the current frame sample image based on the spatiotemporal features of the current frame sample image and the current frame sample image; obtaining the spatiotemporal features of the adjacent frame sample image, and obtaining a fused image feature corresponding to the current frame sample image based on the spatiotemporal features of the adjacent frame sample image, the spatiotemporal features of the current frame sample image, and the spatiotemporal feature weights of the adjacent frame sample image relative to the current frame sample image; inputting the fused image feature corresponding to the current frame sample image into the pointer recognition model to be trained to obtain a predicted result of the pointer position of the pointer instrument to be identified in the current frame sample image; obtaining a loss value based on the difference between the pointer position prediction result and the actual pointer position marked in the current frame sample image, and using the loss value to train the pointer recognition model.
[0122] Each module in the pointer recognition device 900 for a pointer instrument can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.
[0123] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 8 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a pointer recognition method for a pointer instrument is implemented.
[0124] Those skilled in the art will understand that Figure 8The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.
[0125] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.
[0126] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.
[0127] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.
[0128] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), quantum computing-based data processing logic devices, artificial intelligence (AI) processors, and the like.
[0129] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.
[0130] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.
Claims
1. A pointer identification method for a pointer instrument, characterized in that: The method comprises: Obtaining the current frame image and adjacent frame images of the pointer instrument to be identified; Obtaining spatiotemporal features of the current frame image according to the current frame image and the adjacent frame images; Obtaining, based on the spatiotemporal features of the current frame image and the current frame image, spatiotemporal feature weights of the adjacent frame images relative to the current frame image; Acquiring the spatiotemporal features of the adjacent frame images, and obtaining fused image features corresponding to the current frame image based on the spatiotemporal features of the adjacent frame images, the spatiotemporal features of the current frame image, and the spatiotemporal feature weights; The fused image features are input into a pre-trained pointer recognition model to obtain a pointer recognition result of the pointer-type instrument to be recognized in the current frame image.
2. The method according to claim 1, characterized in that The obtaining, according to the current frame image and the adjacent frame images, the spatiotemporal features of the current frame image includes: Acquire current frame visual features corresponding to the current frame image and adjacent frame visual features corresponding to the adjacent frame images; Using a bidirectional attention mechanism, obtaining a current frame attention feature corresponding to the current frame visual feature and an adjacent frame attention feature corresponding to the adjacent frame visual feature; Extracting the temporal relationship between the current frame attention feature and the adjacent frame attention feature through a convolutional gated recurrent unit; According to the temporal relationship and the attention features of the current frame, the spatiotemporal features of the current frame image are obtained.
3. The method according to claim 2, characterized in that The bidirectional attention mechanism includes a spatial attention mechanism and a channel attention mechanism; The method of using a bidirectional attention mechanism to obtain a current frame attention feature corresponding to the current frame visual feature includes: Utilizing a spatial attention mechanism, obtaining a spatial attention feature corresponding to the visual feature of the current frame; Utilizing the channel attention mechanism, obtaining the channel attention feature corresponding to the visual feature of the current frame; Based on the product result of the spatial attention feature and the channel attention feature and the spatial attention feature, the current frame attention feature corresponding to the current frame visual feature is obtained.
4. The method according to claim 2, characterized in that The acquiring, based on the spatiotemporal features of the current frame image and the current frame image, spatiotemporal feature weights of the adjacent frame images relative to the current frame image includes: Performing a splicing operation on the spatiotemporal features of the current frame image and the current frame visual features corresponding to the current frame image in a channel dimension to obtain a splicing feature; The splicing features are subjected to a stacked convolution transformation operation to obtain spatiotemporal feature weights of the adjacent frame images relative to the current frame image.
5. The method according to claim 1, wherein The obtaining, according to the spatiotemporal features of the adjacent frame images, the spatiotemporal features of the current frame image, and the spatiotemporal feature weights, of the fused image features corresponding to the current frame image includes: The spatiotemporal features of the adjacent frame images and the spatiotemporal features of the current frame image are weighted and summed according to the spatiotemporal feature weights to obtain a fused image feature corresponding to the current frame image.
6. The method according to claim 1, wherein The training process of the pointer recognition model includes: Acquiring a visual image of a training pointer instrument and marking the actual pointer position in the visual image; Acquire a current frame sample image and adjacent frame sample images of the training pointer instrument from the visual image; Obtaining spatiotemporal features of the current frame sample image according to the current frame sample image and the adjacent frame sample images; Obtaining, according to the spatiotemporal features of the current frame sample image and the current frame sample image, spatiotemporal feature weights of the adjacent frame sample images relative to the current frame sample image; Acquiring spatiotemporal features of adjacent frame sample images, and obtaining fused image features corresponding to the current frame sample image based on the spatiotemporal features of the adjacent frame sample images, the spatiotemporal features of the current frame sample image, and the spatiotemporal feature weights of the adjacent frame sample images relative to the current frame sample image; Inputting the fused image features corresponding to the current frame sample image into the pointer recognition model to be trained, and obtaining a pointer position prediction result of the pointer instrument to be identified in the current frame sample image; A loss value is obtained according to a difference between the pointer position prediction result and the actual pointer position marked in the current frame sample image, and the pointer recognition model is trained using the loss value.
7. A pointer identification device for a pointer instrument, characterized in that: The device comprises: An image acquisition module is used to acquire a current frame image and adjacent frame images of the pointer instrument to be identified; a feature fusion module, configured to obtain, based on the current frame image and the adjacent frame images, the spatiotemporal features of the current frame image; obtain, based on the spatiotemporal features of the current frame image and the current frame image, the spatiotemporal feature weights of the adjacent frame images relative to the current frame image; obtain the spatiotemporal features of the adjacent frame images, and obtain, based on the spatiotemporal features of the adjacent frame images, the spatiotemporal features of the current frame image, and the spatiotemporal feature weights, the fused image features corresponding to the current frame image; The pointer recognition module is used to input the fused image features into a pre-trained pointer recognition model to obtain a pointer recognition result of the pointer-type instrument to be identified in the current frame image.
8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.
10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.