Reading recognition method and device, network training method and device, equipment and medium

Abstract

This application provides a reading recognition method and apparatus, a network training method and apparatus, equipment, and medium, relating to the field of image digital recognition technology. In this application, firstly, an initial digital instrument image can be segmented to obtain a corresponding target digital instrument image; secondly, a first feature mining process can be performed on the target digital instrument image to obtain corresponding intermediate image features; then, a second feature mining process can be performed on the intermediate image features to obtain corresponding target image features; finally, semantic recognition processing can be performed on the target image features to obtain the target semantic recognition result corresponding to the target digital instrument. Based on the above, the relatively poor reliability of digital instrument reading recognition in the prior art can be improved.

Description

Technical Field

[0001] This application relates to the field of image digital technology, and more specifically, to a reading recognition method and apparatus, a network training method and apparatus, equipment and medium. Background Technology

[0002] Digital instruments are widely used in industrial production to display important parameters such as temperature, pressure, and flow rate. In industrial manufacturing processes, large amounts of real-time production data are needed for purposes such as quality control, production monitoring, and equipment status analysis. Digital instruments are among the most common types of data display instruments. The accuracy and real-time performance of recognizing the numbers displayed by digital instruments directly impact the level of informatization and intelligence in industrial production. Intelligent recognition by digital instruments can effectively solve the risk of misoperation associated with manual reading of instrument data, improve data accuracy, and meet the real-time requirements of production monitoring. However, the inventors have discovered that existing technologies suffer from relatively poor reliability in recognizing digital instrument readings. Summary of the Invention

[0003] In view of this, the purpose of this application is to provide a reading recognition method and apparatus, a network training method and apparatus, a device and a medium, to improve the problem of relatively poor reliability of digital instrument reading recognition in the prior art.

[0004] To achieve the above objectives, this application adopts the following technical solution:

[0005] A method for recognizing digital instrument readings, comprising:

[0006] The initial digital instrument image is segmented to obtain the corresponding target digital instrument image, wherein the initial digital instrument image is formed by performing an image acquisition operation on the target digital instrument, and the target digital instrument image includes at least the digital display area of ​​the target digital instrument;

[0007] The target digital instrument image is subjected to a first feature mining process to obtain corresponding intermediate image features, wherein the intermediate image features are used to characterize the semantic information of the numbers in the target digital instrument image, and the first feature mining process includes at least channel attention processing to mine the semantic information of salient channels in the target digital instrument image.

[0008] The intermediate image features are subjected to a second feature mining process to obtain the corresponding target image features. The second feature mining process is different from the first feature mining process, and the second feature mining process includes at least a correlation feature mining process, which is used to mine the semantic information of the correlation between each feature column in the intermediate image features.

[0009] The target image features are subjected to semantic recognition processing to obtain the target semantic recognition result corresponding to the target digital instrument, wherein the target semantic recognition result includes at least the numbers displayed in the digital display area of ​​the target digital instrument.

[0010] In a preferred embodiment of this application, the step of performing a first feature mining process on the target digital instrument image to obtain corresponding intermediate image features in the above-described digital instrument reading recognition method includes:

[0011] The target digital instrument reading recognition network includes a convolutional network model to perform feature extraction processing on the target digital instrument image, thereby obtaining the corresponding initial image features;

[0012] The target digital instrument reading recognition network includes a channel attention network model, which performs channel attention processing on the initial image features to obtain the corresponding intermediate image features.

[0013] In a preferred embodiment of this application, in the above-described digital instrument reading recognition method, the step of performing channel attention processing on the initial image features using the channel attention network model included in the target digital instrument reading recognition network to obtain the corresponding intermediate image features includes:

[0014] The first pooling unit and the second pooling unit in the channel attention network model included in the target digital instrument reading recognition network are used to perform pooling processing on the initial image features to obtain corresponding first pooling image features and second pooling image features. The first pooling unit and the second pooling unit have different pooling processing methods to capture different information in the initial image features.

[0015] The multilayer perceptron in the channel attention network model included in the target digital instrument reading recognition network is used to map the first pooling image features and the second pooling image features respectively to obtain the corresponding first mapped image features and second mapped image features.

[0016] Based on the first mapped image features and the second mapped image features, the intermediate image features corresponding to the initial image features are determined.

[0017] In a preferred embodiment of this application, in the above-described digital instrument reading recognition method, the step of determining the intermediate image features corresponding to the initial image features based on the first mapped image features and the second mapped image features includes:

[0018] The first mapped image feature and the second mapped image feature are subjected to a first feature fusion process to obtain a corresponding fused mapped image feature, and the fused mapped image feature is subjected to an activation process to obtain a corresponding activated mapped image feature, wherein the activated mapped image feature carries semantic information representing the saliency of image channels;

[0019] The activation mapping image features and the initial image features are subjected to a second feature fusion process to obtain the intermediate image features corresponding to the initial image features.

[0020] In a preferred embodiment of this application, the step of performing a second feature mining process on the intermediate image features to obtain the corresponding target image features in the above-described digital instrument reading recognition method includes:

[0021] The self-attention network model included in the target digital instrument reading recognition network is used to map the intermediate image features to obtain the corresponding query image features, key image features and value image features. Based on the query image features, key image features and value image features, the self-attention image features corresponding to the intermediate image features are determined.

[0022] Using the bidirectional feature capture network model included in the target digital instrument reading recognition network, the self-attention image features are subjected to bidirectional contextual semantic information capture processing to obtain the corresponding first contextual semantic features and second contextual semantic features. Furthermore, the first contextual semantic features and the second contextual semantic features are subjected to feature fusion processing to obtain the target image features corresponding to the self-attention image features.

[0023] This application also provides a method for training a digital instrument reading recognition network, including:

[0024] The initial neural network includes an initial image segmentation network and an initial digital instrument reading recognition network. A first sample digital instrument image, first image label information corresponding to the first sample digital instrument image, a second sample digital instrument image, and second image label information corresponding to the second sample digital instrument image are obtained. The first image label information is used to indicate the digital display area in the first sample digital instrument image, and the second image label information is used to indicate the number displayed in the digital display area in the second sample digital instrument image.

[0025] Using the initial image segmentation network, the first sample digital instrument image is segmented to obtain the corresponding target sample digital instrument image. Based on the target sample digital instrument image and the first image label information, the network parameters of the initial image segmentation network are updated to obtain the corresponding target image segmentation network. The target sample digital instrument image includes at least the digital display area of ​​the digital instrument.

[0026] Using the initial digital instrument reading recognition network, the second sample digital instrument image is subjected to a first feature mining process to obtain the corresponding sample intermediate image features. The sample intermediate image features are used to characterize the semantic information of the numbers in the second sample digital instrument image. The first feature mining process includes at least channel attention processing to mine the semantic information of the salient channels in the second sample digital instrument image.

[0027] Using the initial digital instrument reading recognition network, the intermediate image features of the sample are subjected to a second feature mining process to obtain the corresponding target image features of the sample. The second feature mining process is different from the first feature mining process, and the second feature mining process includes at least a correlation feature mining process, which is used to mine the semantic information of the correlation between each feature column in the intermediate image features of the sample.

[0028] Using the initial digital instrument reading recognition network, semantic recognition processing is performed on the sample target image features to obtain the sample semantic recognition result corresponding to the digital instrument, wherein the sample semantic recognition result includes at least the numbers displayed in the digital display area of ​​the digital instrument;

[0029] Based on the difference between the sample semantic recognition result and the second image label information, the network parameters of the initial digital instrument reading recognition network are updated to obtain the corresponding target digital instrument reading recognition network.

[0030] Based on the target image segmentation network and the target digital instrument reading recognition network, a corresponding target neural network is determined, wherein the target neural network is used to implement each step in the above-described digital instrument reading recognition method.

[0031] In a preferred embodiment of this application, in the above-described digital instrument reading recognition network training method, the steps of segmenting the first sample digital instrument image using the initial image segmentation network to obtain the corresponding target sample digital instrument image, and updating the network parameters of the initial image segmentation network based on the target sample digital instrument image and the first image label information to obtain the corresponding target image segmentation network, include:

[0032] Based on multiple first image label information of multiple first sample digital images, multiple corresponding bounding boxes are determined, wherein each bounding box corresponds to one first image label information, and the bounding box is used to indicate the corresponding digital display area.

[0033] Clustering is performed on the plurality of bounding boxes to obtain corresponding clustering results, and at least one target bounding box is determined based on the clustering results;

[0034] Using the initial image segmentation network, the first sample digital instrument image is segmented based on the at least one target bounding box to obtain the corresponding target sample digital instrument image;

[0035] Based on the target sample digital instrument image and the first image label information, the network parameters of the initial image segmentation network are updated to obtain the corresponding target image segmentation network.

[0036] In a preferred embodiment of this application, in the above-described digital instrument reading recognition network training method, the step of using the initial digital instrument reading recognition network to perform a first feature mining process on the second sample digital instrument image to obtain the corresponding sample intermediate image features includes:

[0037] Using the convolutional network model included in the initial digital instrument reading recognition network, feature extraction processing is performed on the second sample digital instrument image to obtain the corresponding sample image features;

[0038] The first pooling unit and the second pooling unit in the channel attention network model included in the initial digital instrument reading recognition network are used to perform pooling processing on the sample image features to obtain the corresponding sample first pooling image features and sample second pooling image features. The first pooling unit and the second pooling unit have different pooling processing methods to capture different information in the sample image features.

[0039] Using the multilayer perceptron in the channel attention network model, the first pooling image features and the second pooling image features of the sample are mapped to obtain the corresponding first mapped image features and second mapped image features of the sample.

[0040] Based on the first mapped image features and the second mapped image features of the sample, the intermediate image features of the sample corresponding to the sample image features are determined.

[0041] This application also provides a digital instrument reading recognition device, including:

[0042] An image segmentation module is used to segment an initial digital instrument image to obtain a corresponding target digital instrument image. The initial digital instrument image is formed by performing an image acquisition operation on the target digital instrument, and the target digital instrument image includes at least the digital display area of ​​the target digital instrument.

[0043] The first feature mining module is used to perform a first feature mining process on the target digital instrument image to obtain corresponding intermediate image features. The intermediate image features are used to characterize the semantic information of the numbers in the target digital instrument image. The first feature mining process includes at least channel attention processing, which is used to mine the semantic information of the salient channels in the target digital instrument image.

[0044] The second feature mining module is used to perform second feature mining processing on the intermediate image features to obtain the corresponding target image features. The second feature mining processing is different from the first feature mining processing, and the second feature mining processing includes at least correlation feature mining processing, which is used to mine the semantic information of the correlation between each feature column in the intermediate image features.

[0045] A semantic recognition module is used to perform semantic recognition processing on the target image features to obtain the target semantic recognition result corresponding to the target digital instrument, wherein the target semantic recognition result includes at least the numbers displayed in the digital display area of ​​the target digital instrument.

[0046] This application also provides a digital instrument reading recognition network training device, comprising:

[0047] The network acquisition module is used to acquire the initial image segmentation network and the initial digital instrument reading recognition network included in the initial neural network, and to acquire the first sample digital instrument image, the first image label information corresponding to the first sample digital instrument image, the second sample digital instrument image, and the second image label information corresponding to the second sample digital instrument image, wherein the first image label information is used to indicate the digital display area in the first sample digital instrument image, and the second image label information is used to indicate the number displayed in the digital display area in the second sample digital instrument image;

[0048] The first network update module is used to segment the first sample digital instrument image using the initial image segmentation network to obtain a corresponding target sample digital instrument image, and to update the network parameters of the initial image segmentation network based on the target sample digital instrument image and the first image label information to obtain a corresponding target image segmentation network, wherein the target sample digital instrument image includes at least the digital display area of ​​the digital instrument.

[0049] The first sample feature mining module is used to perform a first feature mining process on the second sample digital instrument image using the initial digital instrument reading recognition network to obtain the corresponding sample intermediate image features. The sample intermediate image features are used to characterize the semantic information of the numbers in the second sample digital instrument image, and the first feature mining process includes at least channel attention processing to mine the semantic information of the salient channels in the second sample digital instrument image.

[0050] The second sample feature mining module is used to perform second feature mining processing on the sample intermediate image features using the initial digital instrument reading recognition network to obtain the corresponding sample target image features. The second feature mining processing is different from the first feature mining processing, and the second feature mining processing includes at least correlation feature mining processing, which is used to mine the semantic information of the correlation between each feature column in the sample intermediate image features.

[0051] The sample semantic recognition module is used to perform semantic recognition processing on the sample target image features using the initial digital instrument reading recognition network to obtain the sample semantic recognition result corresponding to the digital instrument, wherein the sample semantic recognition result includes at least the numbers displayed in the digital display area of ​​the digital instrument;

[0052] The second network update module is used to update the network parameters of the initial digital instrument reading recognition network based on the difference between the sample semantic recognition result and the second image label information, so as to obtain the corresponding target digital instrument reading recognition network.

[0053] The target neural network determination module is used to determine the corresponding target neural network based on the target image segmentation network and the target digital instrument reading recognition network.

[0054] Based on the above, this application also provides an electronic device, including:

[0055] Memory, used to store computer programs;

[0056] A processor connected to the memory is used to execute the computer program stored in the memory to implement the above-described digital instrument reading recognition method, or to implement the above-described digital instrument reading recognition network training method.

[0057] Based on the above, this application also provides a computer-readable storage medium storing a computer program that, when executed, performs each step of the above-described digital instrument reading recognition method or digital instrument reading recognition network training method.

[0058] The reading recognition method and apparatus, network training method and apparatus, device, and medium provided in this application firstly segment an initial digital instrument image to obtain a corresponding target digital instrument image; secondly, perform a first feature mining process on the target digital instrument image to obtain corresponding intermediate image features; then, perform a second feature mining process on the intermediate image features to obtain corresponding target image features; finally, perform semantic recognition processing on the target image features to obtain the target semantic recognition result corresponding to the target digital instrument. Based on the above, since the target digital instrument image undergoes two different feature mining processes, the semantic representation ability of the mined target image features is better, thereby improving the reliability of semantic recognition based on target image features. Furthermore, since the first feature mining process can mine the semantic information of salient channels in the target digital instrument image, it can capture more detailed information that is more representative of digital display, thus further improving the reliability of semantic recognition. In addition, since the second feature mining process can mine the semantic information of the correlation between feature columns in the intermediate image features, it can further improve the semantic representation ability of the mined target image features. Based on this, by adopting the improved solution of this application, the reliability of semantic recognition can be fully guaranteed, thereby improving the problem of relatively poor reliability of digital instrument reading recognition in the prior art. Attached Figure Description

[0059] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings.

[0060] Figure 1 A structural block diagram of an electronic device provided in an embodiment of this application.

[0061] Figure 2 This is a flowchart illustrating the digital instrument reading recognition method provided in an embodiment of this application.

[0062] Figure 3 This is a schematic diagram illustrating image segmentation processing as provided in an embodiment of this application.

[0063] Figure 4 This is a comparative schematic diagram of the image channels provided in the embodiments of this application.

[0064] Figure 5 This is a schematic diagram of the first feature mining process provided in an embodiment of this application.

[0065] Figure 6 This is a flowchart illustrating the digital instrument reading recognition network training method provided in an embodiment of this application.

[0066] Figure 7A block diagram of a digital instrument reading recognition device provided in an embodiment of this application.

[0067] Figure 8 This is a block diagram of a digital instrument reading recognition network training device provided in an embodiment of this application. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0069] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0070] like Figure 1 As shown in the figure, this application provides an electronic device. The electronic device may include a memory, a processor, and a target device, wherein the target device may be a digital instrument reading recognition device or a digital instrument reading recognition network training device.

[0071] Specifically, the memory and the processor are electrically connected directly or indirectly to enable data transmission or interaction. For example, the memory and the processor can be electrically connected via one or more communication buses or signal lines. The target device includes at least one software functional module stored in the memory in the form of software or firmware. The processor is used to execute executable computer programs stored in the memory, such as the software functional modules and computer programs included in the target device, to implement the target method provided in the embodiments of this application (such as a digital instrument reading recognition method or a digital instrument reading recognition network training method).

[0072] Optionally, the memory may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0073] Furthermore, the processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), a system on chip (SoC), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0074] Understandable. Figure 1 The structure shown is for illustrative purposes only; the electronic device may also include components that are more advanced than those shown. Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown may include, for example, a communication unit for exchanging information with other devices (such as image acquisition devices).

[0075] Combination Figure 2 This application also provides a digital instrument reading recognition method applicable to the aforementioned electronic device. The method steps defined in the process of the digital instrument reading recognition method can be implemented by the electronic device.

[0076] The following will be about Figure 2 The specific process shown will be explained in detail.

[0077] Step S110: Segment the initial digital instrument image to obtain the corresponding target digital instrument image.

[0078] In this embodiment, the electronic device can perform segmentation processing (also called target detection) on an initial digital instrument image to obtain a corresponding target digital instrument image. The initial digital instrument image is formed by acquiring an image of the target digital instrument, and the target digital instrument image at least includes the digital display area of ​​the target digital instrument. That is, a portion of the image corresponding to the digital display area (e.g., ...) can be detected from the initial digital instrument image. Figure 3 (as shown), to avoid interference from images of other areas in subsequent reading recognition.

[0079] Step S120: Perform first feature mining processing on the target digital instrument image to obtain the corresponding intermediate image features.

[0080] In this embodiment, after obtaining the target digital instrument image, the electronic device can perform a first feature mining process on the target digital instrument image to obtain corresponding intermediate image features. The intermediate image features are used to characterize the semantic information of the numbers in the target digital instrument image, and the first feature mining process includes at least channel attention processing to mine the semantic information of salient channels in the target digital instrument image, such as... Figure 4 As shown, the inventors of this application have discovered through research that the characteristics of different channels (color channels) have different levels of importance (salience) for digital display. Therefore, through channel attention processing, important channel information can be given special attention. For example, the R channel has a stronger representational ability, so the R channel can be emphasized.

[0081] Step S130: Perform second feature mining processing on the intermediate image features to obtain the corresponding target image features.

[0082] In this embodiment, after obtaining the intermediate image features, the electronic device can perform a second feature mining process on the intermediate image features to obtain the corresponding target image features. The second feature mining process differs from the first feature mining process, and includes at least a correlation feature mining process to mine the semantic information of the correlation between the feature columns in the intermediate image features. That is, the intermediate image features can include multiple feature columns, thus allowing the mining of the correlation information between the feature columns.

[0083] Step S140: Perform semantic recognition processing on the target image features to obtain the target semantic recognition result corresponding to the target digital instrument.

[0084] In this embodiment of the application, after obtaining the target image features, the electronic device can perform semantic recognition processing on the target image features (such as recognizing the corresponding numbers) to obtain the target semantic recognition result corresponding to the target digital instrument. The target semantic recognition result includes at least the numbers displayed in the digital display area of ​​the target digital instrument.

[0085] Based on the above, since the target digital instrument image undergoes two different feature mining processes, the semantic representation capability of the mined target image features is improved, thereby enhancing the reliability of semantic recognition based on target image features. Furthermore, because the first feature mining process can mine the semantic information of salient channels in the target digital instrument image, it can capture more representative details of the digital display, thus further improving the reliability of semantic recognition. Additionally, because the second feature mining process can mine the semantic information of the correlation between feature columns in the intermediate image features, it can further improve the semantic representation capability of the mined target image features. Therefore, by adopting the improved solution of this application, the reliability of semantic recognition can be fully guaranteed, thereby improving the relatively poor reliability of digital instrument reading recognition in the prior art.

[0086] Firstly, regarding step S110, it should be noted that the specific method for segmenting the initial digital instrument image is not limited and can be selected according to actual needs.

[0087] For example, in an alternative implementation, to improve the efficiency and accuracy of segmenting the initial digital instrument image, a neural network can be used for target detection. Then, the detected digital display area is segmented to obtain the corresponding target digital instrument image. Exemplarily, the YOLO series of models (such as YOLO5) can be used for target detection. The core idea of ​​the YOLO series of models is to directly predict bounding boxes and categories in the image using a single neural network model, thereby achieving real-time target detection.

[0088] Secondly, regarding step S120, it should be noted that the specific method of performing the first feature mining processing on the target digital instrument image is not limited and can be selected according to actual needs.

[0089] For example, in an alternative implementation, in order to improve the reliability of the first feature mining process on the target digital instrument image, the above step S120 may further include steps S121 and S122, the specific contents of each step of which are described below.

[0090] Step S121: Using the convolutional network model included in the target digital instrument reading recognition network, feature extraction processing is performed on the target digital instrument image to obtain the corresponding initial image features.

[0091] In this embodiment of the application, the target digital instrument reading recognition network can first use a convolutional network model (such as a convolutional neural network (CNN), which may include a convolutional layer to extract features of the target digital instrument image through convolution operations) to perform feature extraction processing on the target digital instrument image to obtain the corresponding initial image features. In this way, shallow image features can be obtained.

[0092] Step S122: Using the channel attention network model included in the target digital instrument reading recognition network, channel attention processing is performed on the initial image features to obtain the corresponding intermediate image features.

[0093] In this embodiment of the application, after obtaining the initial image features, the channel attention network model included in the target digital instrument reading recognition network can be further used to perform channel attention processing on the initial image features to obtain the corresponding intermediate image features.

[0094] It is understood that in step S122 above, the specific method of performing channel attention processing on the initial image features is not limited and can be selected according to actual needs.

[0095] For example, in an alternative implementation, in order to make the mined intermediate image features have better representational ability, the above step S122 may further include steps S122a, S122b and S122c, the specific contents of each step are as follows.

[0096] Step S122a: Using the first pooling unit and the second pooling unit in the channel attention network model included in the target digital instrument reading recognition network, the initial image features are pooled respectively to obtain the corresponding first pooling image features and second pooling image features.

[0097] In the embodiments of this application, combined with Figure 5The initial image features can be pooled using the first and second pooling units in the channel attention network model of the target digital instrument reading recognition network, respectively, to obtain corresponding first and second pooled image features. For example, the first pooling unit is used to pool the initial image features to obtain the first pooled image features; the second pooling unit is used to pool the initial image features to obtain the second pooled image features. The first and second pooling units have different pooling methods to capture different information in the initial image features. For example, the first pooling unit can be a max pooling unit, which can obtain important information from the image and remove interference information; the second pooling unit can be an average pooling unit, which can remove some unimportant details and help the model generate a smoother output, thereby improving the model's generalization ability.

[0098] Step S122b: Using the multilayer perceptron in the channel attention network model included in the target digital instrument reading recognition network, the first pooling image features and the second pooling image features are mapped to obtain the corresponding first mapped image features and second mapped image features.

[0099] In this embodiment, after obtaining the first pooling image features and the second pooling image features, a multi-layer perceptron (MLP) in the channel attention network model included in the target digital instrument reading recognition network can be used to map the first pooling image features and the second pooling image features respectively, to obtain a first mapped image feature and a second mapped image feature. For example, the first pooling image features can be processed by the MLP to obtain the first mapped image feature, and the second pooling image features can be processed by the MLP to obtain the corresponding second mapped image feature.

[0100] Step S122c: Based on the first mapped image features and the second mapped image features, determine the intermediate image features corresponding to the initial image features.

[0101] In this embodiment of the application, after obtaining the first mapped image features and the second mapped image features, the intermediate image features corresponding to the initial image features can be determined based on the first mapped image features and the second mapped image features.

[0102] It should be noted that the specific method for determining the intermediate image features based on the first mapped image features and the second mapped image features is not limited and can be selected according to actual needs.

[0103] For example, in an alternative implementation, in order to ensure that the obtained intermediate image features can adequately represent the feature information of each channel to different degrees, thereby improving the representational capability of the intermediate image features, the above step S122c may further include the following:

[0104] First, a first feature fusion process can be performed on the first mapped image features and the second mapped image features to obtain corresponding fused mapped image features. Then, activation processing can be performed on the fused mapped image features to obtain corresponding activated mapped image features. The activated mapped image features carry semantic information representing the saliency of image channels. For example, the first mapped image features and the second mapped image features can be superimposed to obtain corresponding fused mapped image features. Then, activation processing can be performed on the fused mapped image features using a sigmoid function or similar method to obtain corresponding activated mapped image features. This allows the activated mapped image features to represent the weight values ​​corresponding to each channel. It should be noted that in other embodiments, the first mapped image features and the second mapped image features can also be activated separately using a sigmoid function or similar method to obtain first activated features (such as...). Figure 5 The alpha1 shown) and the second activation feature (such as Figure 5 (as shown in alpha2), then, the first activation feature and the second activation feature are superimposed to obtain the activation mapping image features (such as alpha2). Figure 5 (Alpha shown).

[0105] Secondly, a second feature fusion process can be performed on the activation mapping image features and the initial image features to obtain intermediate image features corresponding to the initial image features. For example, the activation mapping image features and the initial image features can be directly superimposed to obtain the corresponding intermediate image features, or the activation mapping image features and the initial image features can be weighted and superimposed to obtain the corresponding intermediate image features. The weighting coefficients of the weighted superposition can be used as network parameters of the corresponding neural network and updated during training. Based on this, the obtained intermediate image features can focus more on the relatively more important channel information.

[0106] Thirdly, regarding step S130, it should be noted that the specific method of performing the second feature mining processing on the intermediate image features is not limited and can be selected according to actual needs.

[0107] For example, in an alternative implementation, in order to further improve the representational ability of the obtained target image features, the above step S130 may further include steps S131 and S132, the specific contents of each step are as follows.

[0108] Step S131: Using the self-attention network model included in the target digital instrument reading recognition network, the intermediate image features are mapped to obtain the corresponding query image features, key image features and value image features. Based on the query image features, key image features and value image features, the self-attention image features corresponding to the intermediate image features are determined.

[0109] In this embodiment, the self-attention network model included in the target digital instrument reading recognition network can be used to map the intermediate image features to obtain corresponding query image features, key image features, and value image features. Based on the query image features, key image features, and value image features, the self-attention image features corresponding to the intermediate image features are determined. For example, the self-attention network model may include three matrices, such as a query matrix, a key matrix, and a value matrix. Then, based on the query matrix, key matrix, and value matrix, the intermediate image features can be mapped (e.g., multiplied) to obtain the corresponding query image features, key image features, and value image features. For instance, a similarity matrix between the query image features and the key image features can be calculated, and then the similarity matrix and the value image features can be multiplied (e.g., multiplied using matrix multiplication) to obtain the corresponding self-attention image features.

[0110] Step S132: Using the bidirectional feature capture network model included in the target digital instrument reading recognition network, the self-attention image features are subjected to bidirectional contextual semantic information capture processing to obtain the corresponding first contextual semantic features and second contextual semantic features. Then, the first contextual semantic features and the second contextual semantic features are subjected to feature fusion processing to obtain the target image features corresponding to the self-attention image features.

[0111] In this embodiment, after obtaining the self-attention image features, the bidirectional feature capture network model (such as a bidirectional long short-term memory network with a forgetting mechanism, which may include two long short-term memory networks in opposite directions, one processing the input sequence in forward order and the other processing the input sequence in reverse order; each direction of the long short-term memory network generates an output sequence, and these output sequences are merged or connected together to provide a comprehensive view of bidirectional information) can be used to capture bidirectional contextual semantic information of the self-attention image features to obtain the corresponding first contextual semantic features and second contextual semantic features. Furthermore, feature fusion processing is performed on the first contextual semantic features and the second contextual semantic features to obtain the target image features corresponding to the self-attention image features.

[0112] Fourthly, regarding step S140, it should be noted that the specific method of semantic recognition processing of the target image features is not limited and can be selected according to actual needs.

[0113] For example, in an alternative implementation, to ensure higher reliability of semantic recognition processing and obtain reliable target semantic recognition results, character prediction output can be performed based on each feature in the target image features. For instance, the character with the highest probability among all characters corresponding to that feature can be used as the output (prediction result) of that feature. Additionally, duplicate characters and placeholders can be removed to obtain the corresponding text recognition content (number). For example, semantic recognition processing of the target image features can be performed using the transcription layer included in the target digital instrument reading recognition network. That is, the transcription layer can map serialized features to sequences of text or labels. The transcription layer can consist of one or more fully connected layers used to map the target image features to a label space. These fully connected layers can generate a probability distribution of the output sequence using activation functions (such as softmax).

[0114] Combination Figure 6 This application also provides a method for training a digital instrument reading recognition network that can be applied to the aforementioned electronic device. The method steps defined in the process of the digital instrument reading recognition network training method can be implemented by the electronic device.

[0115] The following will be about Figure 7 The specific process shown will be explained in detail.

[0116] Step S210: Obtain the initial image segmentation network and the initial digital instrument reading recognition network included in the initial neural network, and obtain the first sample digital instrument image, the first image label information corresponding to the first sample digital instrument image, the second sample digital instrument image, and the second image label information corresponding to the second sample digital instrument image.

[0117] In this embodiment, the electronic device can first acquire an initial image segmentation network and an initial digital instrument reading recognition network, including an initial neural network, and acquire a first sample digital instrument image, first image label information corresponding to the first sample digital instrument image, a second sample digital instrument image, and second image label information corresponding to the second sample digital instrument image. The first image label information is used to indicate the digital display area in the first sample digital instrument image (e.g., marked by a rectangle), and the second image label information is used to indicate the digit displayed in the digital display area of ​​the second sample digital instrument image. Furthermore, the first sample digital instrument image and the second sample digital instrument image can be the same or different. For example, to improve the robustness of the network model, image enhancement (for the first sample digital instrument image) can be achieved through image rotation, cropping, filtering, histogram equalization, etc., which can effectively enhance image contrast and address issues such as uneven lighting. The processed image is then labeled with rectangles corresponding to the "digital display area," retaining the corresponding width, height, coordinates, etc. Then, the initial neural network (e.g., a YOLO5 model) can be trained using the labeled dataset, and each image is segmented based on the labeled rectangles, i.e., the image of the digital display area is saved. The segmented digital display area image can be used as input for subsequent models (e.g., as a second sample digital instrument image).

[0118] Step S220: Using the initial image segmentation network, the first sample digital instrument image is segmented to obtain the corresponding target sample digital instrument image; and based on the target sample digital instrument image and the first image label information, the network parameters of the initial image segmentation network are updated to obtain the corresponding target image segmentation network.

[0119] In this embodiment, after obtaining the initial image segmentation network, the first sample digital instrument image, and the corresponding first image label information, the electronic device can use the initial image segmentation network to segment the first sample digital instrument image (e.g., perform target region detection) to obtain the corresponding target sample digital instrument image. Furthermore, based on the difference between the target sample digital instrument image and the first image label information, the network parameters of the initial image segmentation network are updated to obtain the corresponding target image segmentation network. The target sample digital instrument image at least includes the digital display area of ​​the digital instrument.

[0120] Step S230: Using the initial digital instrument reading recognition network, perform first feature mining processing on the second sample digital instrument image to obtain the corresponding sample intermediate image features.

[0121] In this embodiment, after obtaining the initial digital instrument reading recognition network, the second sample digital instrument image, and the corresponding second image label information, the electronic device can utilize the initial digital instrument reading recognition network to perform a first feature mining process on the second sample digital instrument image to obtain corresponding intermediate image features. The intermediate image features are used to characterize the semantic information of the numbers in the second sample digital instrument image, and the first feature mining process includes at least channel attention processing to mine the semantic information of salient channels in the second sample digital instrument image, as described previously.

[0122] Step S240: Using the initial digital instrument reading recognition network, perform second feature mining processing on the intermediate image features of the sample to obtain the corresponding sample target image features.

[0123] In this embodiment, after obtaining the intermediate image features of the sample, the electronic device can utilize the initial digital instrument reading recognition network to perform a second feature mining process on the intermediate image features of the sample to obtain the corresponding target image features of the sample. The second feature mining process differs from the first feature mining process, and the second feature mining process includes at least a correlation feature mining process, used to mine the semantic information of the correlation between the feature columns in the intermediate image features of the sample, as described above.

[0124] Step S250: Using the initial digital instrument reading recognition network, perform semantic recognition processing on the sample target image features to obtain the sample semantic recognition result corresponding to the digital instrument.

[0125] In this embodiment of the application, after obtaining the sample target image features, the electronic device can use the initial digital instrument reading recognition network to perform semantic recognition processing on the sample target image features to obtain the sample semantic recognition result corresponding to the digital instrument. The sample semantic recognition result includes at least the numbers displayed in the digital display area of ​​the digital instrument, as described above.

[0126] Step S260: Based on the difference between the sample semantic recognition result and the second image label information, update the network parameters of the initial digital instrument reading recognition network to obtain the corresponding target digital instrument reading recognition network.

[0127] In this embodiment, after obtaining the sample semantic recognition result, the electronic device can update the network parameters of the initial digital instrument reading recognition network based on the difference between the sample semantic recognition result and the second image label information to obtain the corresponding target digital instrument reading recognition network. For example, the CTC loss (Connectionist Temporal Classification Loss) between the sample semantic recognition result and the second image label information can be calculated. For image text recognition tasks, especially for outputs of variable-length sequences, CTC loss is a commonly used loss function that allows the network to freely align the input and output sequences without requiring precise alignment. In other embodiments, the corresponding loss can also be calculated based on other methods (such as edit distance), and then the network parameters of the initial digital instrument reading recognition network can be updated along the direction of reducing this loss to obtain the corresponding target digital instrument reading recognition network (e.g., the number of updates reaches a preset number, or the loss is reduced to a preset loss, or the magnitude of the loss reduction is less than a preset magnitude, etc.).

[0128] Step S270: Based on the target image segmentation network and the target digital instrument reading recognition network, determine the corresponding target neural network.

[0129] In this embodiment, after obtaining the target image segmentation network and the target digital instrument reading recognition network, a corresponding target neural network can be determined based on these networks. For example, the target image segmentation network and the target digital instrument reading recognition network can be directly combined to form the target neural network, or the corresponding target neural network can be constructed based on the network parameters of the target image segmentation network and the target digital instrument reading recognition network. The target neural network can be used to implement each step in the aforementioned digital instrument reading recognition method.

[0130] Firstly, in step S220 above, the specific method of updating the initial image segmentation network is not limited and can be selected according to actual needs.

[0131] For example, in an alternative implementation, in order to make the updated target image segmentation network more reliable so as to reliably identify and detect the digital display area, the above step S220 may further include steps S221, S222, S223 and S224, the specific contents of each step are as follows.

[0132] Step S221: Based on the multiple first image label information of the multiple first sample digital images, determine the corresponding multiple bounding boxes.

[0133] In this embodiment of the application, after obtaining multiple first sample digital images, multiple corresponding bounding boxes can be determined based on multiple first image label information possessed by the multiple first sample digital images. Each bounding box corresponds to one piece of first image label information, and the bounding box is used to indicate the corresponding digital display area.

[0134] Step S222: Perform clustering processing on the multiple bounding boxes to obtain corresponding clustering results, and determine at least one target bounding box based on the clustering results.

[0135] In this embodiment of the application, after obtaining the plurality of bounding boxes, clustering processing can be performed on the plurality of bounding boxes to obtain corresponding clustering results, and at least one target bounding box can be determined based on the clustering results. For example, the target bounding boxes (anchor boxes) can be obtained based on the K-Means clustering algorithm or other clustering algorithms and used for subsequent training. For instance, the bounding boxes belonging to the cluster centers can be used as target bounding boxes.

[0136] Step S223: Using the initial image segmentation network, the first sample digital instrument image is segmented based on the at least one target bounding box to obtain the corresponding target sample digital instrument image.

[0137] In this embodiment of the application, after obtaining the at least one target bounding box, the initial image segmentation network can be used to segment the first sample digital instrument image based on the at least one target bounding box to obtain the corresponding target sample digital instrument image.

[0138] Step S224: Based on the target sample digital instrument image and the first image label information, update the network parameters of the initial image segmentation network to obtain the corresponding target image segmentation network.

[0139] In this embodiment, after obtaining the target sample digital instrument image, the network parameters of the initial image segmentation network can be updated based on the target sample digital instrument image and the first image label information to obtain the corresponding target image segmentation network. For example, the corresponding error (or loss, the specific loss function is not specifically limited here) can be determined based on the difference between the target sample digital instrument image and the first image label information. Then, the network parameters of the initial image segmentation network can be updated along the direction of reducing this error to obtain the corresponding target image segmentation network. For example, as described above, for a dataset that has been labeled (i.e., multiple labeled images), it can be divided in a 7:2:1 ratio, corresponding to the training set, validation set, and test set, respectively. Then, the YOLO5 model can be trained using the training set data, and the parameters can be updated. The model's performance can be evaluated using the validation set during training. Training stops when the accuracy reaches a specified threshold and the loss function no longer changes. The test set is the dataset used to finally evaluate the model's performance. Once the model has gone through the training and validation phases, evaluation on a test set that has not been used before can more objectively assess the model's generalization ability. The purpose of the test set is to simulate new data encountered by the model in real-world applications in order to more accurately evaluate the model's performance.

[0140] Secondly, in step S230 above, the specific method of performing the first feature mining processing on the second sample digital instrument image is not limited and can be selected according to actual needs.

[0141] For example, in an alternative implementation, in order to enable the mined intermediate image features of the sample to have a better representation ability for the second sample digital instrument image, the above step S230 may further include steps S231, S232, S233 and S234, the specific contents of each step are as follows.

[0142] Step S231: Using the convolutional network model included in the initial digital instrument reading recognition network, feature extraction processing is performed on the second sample digital instrument image to obtain the corresponding sample image features.

[0143] In this embodiment of the application, after obtaining the second sample digital instrument image, the convolutional network model included in the initial digital instrument reading recognition network can be used to perform feature extraction processing on the second sample digital instrument image to obtain the corresponding sample image features. The specific process of feature extraction processing is as described above.

[0144] Step S232: Using the first pooling unit and the second pooling unit in the channel attention network model included in the initial digital instrument reading recognition network, the sample image features are pooled respectively to obtain the corresponding sample first pooling image features and sample second pooling image features.

[0145] In this embodiment, after obtaining the sample image features, the first and second pooling units in the channel attention network model included in the initial digital instrument reading recognition network can be used to perform pooling processing on the sample image features, respectively, to obtain corresponding first pooled image features and second pooled image features. The first and second pooling units have different pooling processing methods to capture different information in the sample image features, as described above.

[0146] Step S233: Using the multilayer perceptron in the channel attention network model, the first pooling image features and the second pooling image features of the sample are mapped to obtain the corresponding first mapped image features and second mapped image features of the sample.

[0147] In this embodiment of the application, after obtaining the first pooling image features and the second pooling image features of the sample, the multilayer perceptron in the channel attention network model can be used to map the first pooling image features and the second pooling image features of the sample respectively to obtain the corresponding first mapped image features and second mapped image features of the sample, as described above.

[0148] Step S234: Based on the first mapped image features of the sample and the second mapped image features of the sample, determine the intermediate image features of the sample corresponding to the sample image features.

[0149] In this embodiment of the application, after obtaining the first mapped image features and the second mapped image features of the sample, the intermediate image features of the sample corresponding to the sample image features can be determined based on the first mapped image features and the second mapped image features of the sample, as described above.

[0150] Combination Figure 7 This application also provides a digital instrument reading recognition device applicable to the aforementioned electronic devices. The digital instrument reading recognition device may include an image segmentation module, a first feature mining module, a second feature mining module, and a semantic recognition module.

[0151] The image segmentation module can be used to segment an initial digital instrument image to obtain a corresponding target digital instrument image. The initial digital instrument image is formed by acquiring an image of the target digital instrument, and the target digital instrument image includes at least the digital display area of ​​the target digital instrument. In this embodiment, the image segmentation module can be used to perform... Figure 2 The relevant content regarding the image segmentation module in step S110 shown can be found in the previous description of step S110.

[0152] The first feature mining module can be used to perform a first feature mining process on the target digital instrument image to obtain corresponding intermediate image features. The intermediate image features are used to characterize the semantic information of the numbers in the target digital instrument image. The first feature mining process includes at least channel attention processing to mine the semantic information of salient channels in the target digital instrument image. In this embodiment, the first feature mining module can be used to execute... Figure 2 The relevant content regarding the first feature mining module in step S120 shown can be found in the previous description of step S120.

[0153] The second feature mining module can be used to perform a second feature mining process on the intermediate image features to obtain corresponding target image features. The second feature mining process differs from the first feature mining process, and the second feature mining process includes at least a correlation feature mining process to mine the semantic information of the correlation between feature columns in the intermediate image features. In this embodiment, the second feature mining module can be used to execute... Figure 2 The relevant content regarding the second feature mining module in step S130 shown can be found in the previous description of step S130.

[0154] The semantic recognition module can be used to perform semantic recognition processing on the target image features to obtain a target semantic recognition result corresponding to the target digital instrument, wherein the target semantic recognition result includes at least the digits displayed in the digital display area of ​​the target digital instrument. In this embodiment, the semantic recognition module can be used to perform... Figure 2 The relevant content regarding the semantic recognition module in step S140 shown can be found in the preceding description of step S140.

[0155] Combination Figure 8This application also provides a digital instrument reading recognition network training device applicable to the aforementioned electronic devices. The digital instrument reading recognition network training device may include a network acquisition module, a first network update module, a first sample feature mining module, a second sample feature mining module, a sample semantic recognition module, a second network update module, and a target neural network determination module.

[0156] The network acquisition module is used to acquire the initial image segmentation network and the initial digital instrument reading recognition network included in the initial neural network, and to acquire a first sample digital instrument image, first image label information corresponding to the first sample digital instrument image, a second sample digital instrument image, and second image label information corresponding to the second sample digital instrument image. The first image label information is used to indicate the digit display area in the first sample digital instrument image, and the second image label information is used to indicate the digit displayed in the digit display area of ​​the second sample digital instrument image. In this embodiment, the network acquisition module can be used to perform... Figure 6 For details regarding the network acquisition module in step S210, please refer to the preceding description of step S210.

[0157] The first network update module is configured to segment the first sample digital instrument image using the initial image segmentation network to obtain a corresponding target sample digital instrument image, and to update the network parameters of the initial image segmentation network based on the target sample digital instrument image and the first image label information to obtain a corresponding target image segmentation network, wherein the target sample digital instrument image includes at least the digital display area of ​​the digital instrument. In this embodiment, the first network update module can be used to perform... Figure 6 For details regarding step S220 shown, please refer to the previous description of step S220 for information about the first network update module.

[0158] The first sample feature mining module is used to perform a first feature mining process on the second sample digital instrument image using the initial digital instrument reading recognition network to obtain corresponding intermediate image features. The intermediate image features are used to characterize the semantic information of the numbers in the second sample digital instrument image, and the first feature mining process includes at least channel attention processing to mine the semantic information of salient channels in the second sample digital instrument image. In this embodiment, the first sample feature mining module can be used to execute... Figure 6 The relevant content regarding the first sample feature mining module in step S230 shown can be found in the previous description of step S230.

[0159] The second sample feature mining module is used to perform second feature mining processing on the intermediate image features of the sample using the initial digital instrument reading recognition network to obtain the corresponding target image features of the sample. The second feature mining processing differs from the first feature mining processing, and the second feature mining processing includes at least correlation feature mining processing, used to mine the semantic information of the correlation between feature columns in the intermediate image features of the sample. In this embodiment, the second sample feature mining module can be used to execute... Figure 6 The relevant content regarding the second sample feature mining module in step S240 is described above.

[0160] The sample semantic recognition module is used to perform semantic recognition processing on the sample target image features using the initial digital instrument reading recognition network, to obtain the sample semantic recognition result corresponding to the digital instrument, wherein the sample semantic recognition result includes at least the digits displayed in the digital display area of ​​the digital instrument. In this embodiment, the sample semantic recognition module can be used to perform... Figure 6 The relevant content regarding the sample semantic recognition module in step S250 shown can be found in the previous description of step S250.

[0161] The second network update module is used to update the network parameters of the initial digital instrument reading recognition network based on the difference between the sample semantic recognition result and the second image label information, thereby obtaining the corresponding target digital instrument reading recognition network. In this embodiment, the second network update module can be used to perform... Figure 6 For details regarding step S260 shown, please refer to the preceding description of step S260 for information about the second network update module.

[0162] The target neural network determination module is used to determine the corresponding target neural network based on the target image segmentation network and the target digital instrument reading recognition network. In this embodiment, the target neural network determination module can be used to perform... Figure 6 The relevant content regarding the target neural network determination module in step S270 shown can be found in the previous description of step S270.

[0163] In this embodiment of the application, corresponding to the above-described digital instrument reading recognition method or digital instrument reading recognition network training method applied to the electronic device, a computer-readable storage medium is also provided. This computer-readable storage medium stores a computer program, which, when executed, performs various steps of the digital instrument reading recognition method and / or the digital instrument reading recognition network training method. The steps executed by the aforementioned computer program are not described in detail here, but can be referred to the preceding explanation of the digital instrument reading recognition method and / or the digital instrument reading recognition network training method.

[0164] In summary, the reading recognition method and apparatus, network training method and apparatus, device, and medium provided in this application firstly segment an initial digital instrument image to obtain a corresponding target digital instrument image; secondly, perform a first feature mining process on the target digital instrument image to obtain corresponding intermediate image features; then, perform a second feature mining process on the intermediate image features to obtain corresponding target image features; and finally, perform semantic recognition processing on the target image features to obtain the target semantic recognition result corresponding to the target digital instrument. Based on the above, since the target digital instrument image undergoes two different feature mining processes, the semantic representation ability of the mined target image features is better, thereby improving the reliability of semantic recognition based on target image features. Furthermore, since the first feature mining process can mine the semantic information of salient channels in the target digital instrument image, it can capture more detailed information that is more representative of digital display, thus further improving the reliability of semantic recognition. In addition, since the second feature mining process can mine the semantic information of the correlation between feature columns in the intermediate image features, it can further improve the semantic representation ability of the mined target image features. Based on this, by adopting the improved solution of this application, the reliability of semantic recognition can be fully guaranteed, thereby improving the problem of relatively poor reliability of digital instrument reading recognition in the prior art.

[0165] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus and method embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0166] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0167] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, electronic device, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks. It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. In the absence of further restrictions, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0168] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for recognizing digital instrument readings, characterized in that, include: The initial digital instrument image is segmented to obtain the corresponding target digital instrument image, wherein the initial digital instrument image is formed by performing an image acquisition operation on the target digital instrument, and the target digital instrument image includes at least the digital display area of ​​the target digital instrument; The target digital instrument image is subjected to a first feature mining process to obtain corresponding intermediate image features, wherein the intermediate image features are used to characterize the semantic information of the numbers in the target digital instrument image, and the first feature mining process includes at least channel attention processing to mine the semantic information of salient channels in the target digital instrument image. The intermediate image features are mapped using the self-attention network model included in the target digital instrument reading recognition network to obtain corresponding query image features, key image features, and value image features. Based on the query image features, key image features, and value image features, the self-attention image features corresponding to the intermediate image features are determined. The bidirectional feature capture network model included in the target digital instrument reading recognition network is used to capture bidirectional contextual semantic information of the self-attention image features to obtain corresponding first contextual semantic features and second contextual semantic features. The first contextual semantic features and the second contextual semantic features are then fused to obtain the target image features corresponding to the self-attention image features. The target image features are subjected to semantic recognition processing to obtain the target semantic recognition result corresponding to the target digital instrument, wherein the target semantic recognition result includes at least the numbers displayed in the digital display area of ​​the target digital instrument.

2. The digital instrument reading recognition method according to claim 1, characterized in that, The step of performing a first feature mining process on the target digital instrument image to obtain the corresponding intermediate image features includes: The target digital instrument reading recognition network includes a convolutional network model to perform feature extraction processing on the target digital instrument image, thereby obtaining the corresponding initial image features; The target digital instrument reading recognition network includes a channel attention network model, which performs channel attention processing on the initial image features to obtain the corresponding intermediate image features.

3. The digital instrument reading recognition method according to claim 2, characterized in that, The step of using the channel attention network model included in the target digital instrument reading recognition network to perform channel attention processing on the initial image features to obtain the corresponding intermediate image features includes: The first pooling unit and the second pooling unit in the channel attention network model included in the target digital instrument reading recognition network are used to perform pooling processing on the initial image features to obtain corresponding first pooling image features and second pooling image features. The first pooling unit and the second pooling unit have different pooling processing methods to capture different information in the initial image features. The multilayer perceptron in the channel attention network model included in the target digital instrument reading recognition network is used to map the first pooling image features and the second pooling image features respectively to obtain the corresponding first mapped image features and second mapped image features. Based on the first mapped image features and the second mapped image features, the intermediate image features corresponding to the initial image features are determined.

4. The digital instrument reading recognition method according to claim 3, characterized in that, The step of determining the intermediate image features corresponding to the initial image features based on the first mapped image features and the second mapped image features includes: The first mapped image feature and the second mapped image feature are subjected to a first feature fusion process to obtain a corresponding fused mapped image feature, and the fused mapped image feature is subjected to an activation process to obtain a corresponding activated mapped image feature, wherein the activated mapped image feature carries semantic information representing the saliency of image channels; The activation mapping image features and the initial image features are subjected to a second feature fusion process to obtain the intermediate image features corresponding to the initial image features.

5. A method for training a digital instrument reading recognition network, characterized in that, include: The initial neural network includes an initial image segmentation network and an initial digital instrument reading recognition network. A first sample digital instrument image, first image label information corresponding to the first sample digital instrument image, a second sample digital instrument image, and second image label information corresponding to the second sample digital instrument image are obtained. The first image label information is used to indicate the digital display area in the first sample digital instrument image, and the second image label information is used to indicate the number displayed in the digital display area in the second sample digital instrument image. Using the initial image segmentation network, the first sample digital instrument image is segmented to obtain the corresponding target sample digital instrument image. Based on the target sample digital instrument image and the first image label information, the network parameters of the initial image segmentation network are updated to obtain the corresponding target image segmentation network. The target sample digital instrument image includes at least the digital display area of ​​the digital instrument. Using the initial digital instrument reading recognition network, the second sample digital instrument image is subjected to a first feature mining process to obtain the corresponding sample intermediate image features. The sample intermediate image features are used to characterize the semantic information of the numbers in the second sample digital instrument image. The first feature mining process includes at least channel attention processing to mine the semantic information of the salient channels in the second sample digital instrument image. Using the initial digital instrument reading recognition network, the intermediate image features of the sample are subjected to a second feature mining process to obtain the corresponding target image features of the sample. The second feature mining process is different from the first feature mining process, and the second feature mining process includes at least a correlation feature mining process, which is used to mine the semantic information of the correlation between each feature column in the intermediate image features of the sample. Using the initial digital instrument reading recognition network, semantic recognition processing is performed on the sample target image features to obtain the sample semantic recognition result corresponding to the digital instrument, wherein the sample semantic recognition result includes at least the numbers displayed in the digital display area of ​​the digital instrument; Based on the difference between the sample semantic recognition result and the second image label information, the network parameters of the initial digital instrument reading recognition network are updated to obtain the corresponding target digital instrument reading recognition network. Based on the target image segmentation network and the target digital instrument reading recognition network, a corresponding target neural network is determined, wherein the target neural network is used to implement each step of the digital instrument reading recognition method according to any one of claims 1-4.

6. The digital instrument reading recognition network training method according to claim 5, characterized in that, The steps of segmenting the first sample digital instrument image using the initial image segmentation network to obtain the corresponding target sample digital instrument image, and updating the network parameters of the initial image segmentation network based on the target sample digital instrument image and the first image label information to obtain the corresponding target image segmentation network, include: Based on multiple first image label information of multiple first sample digital instrument images, multiple corresponding bounding boxes are determined, wherein each bounding box corresponds to one first image label information, and the bounding box is used to indicate the corresponding digital display area. Clustering is performed on the plurality of bounding boxes to obtain corresponding clustering results, and at least one target bounding box is determined based on the clustering results; Using the initial image segmentation network, the first sample digital instrument image is segmented based on the at least one target bounding box to obtain the corresponding target sample digital instrument image; Based on the target sample digital instrument image and the first image label information, the network parameters of the initial image segmentation network are updated to obtain the corresponding target image segmentation network.

7. The digital instrument reading recognition network training method according to claim 5, characterized in that, The step of using the initial digital instrument reading recognition network to perform a first feature mining process on the second sample digital instrument image to obtain the corresponding sample intermediate image features includes: Using the convolutional network model included in the initial digital instrument reading recognition network, feature extraction processing is performed on the second sample digital instrument image to obtain the corresponding sample image features; The first pooling unit and the second pooling unit in the channel attention network model included in the initial digital instrument reading recognition network are used to perform pooling processing on the sample image features to obtain the corresponding sample first pooling image features and sample second pooling image features. The first pooling unit and the second pooling unit have different pooling processing methods to capture different information in the sample image features. Using the multilayer perceptron in the channel attention network model, the first pooling image features and the second pooling image features of the sample are mapped to obtain the corresponding first mapped image features and second mapped image features of the sample. Based on the first mapped image features and the second mapped image features of the sample, the intermediate image features of the sample corresponding to the sample image features are determined.

8. A digital instrument reading recognition device, characterized in that, For implementing the digital instrument reading recognition method according to any one of claims 1-4, the digital instrument reading recognition device comprises: An image segmentation module is used to segment an initial digital instrument image to obtain a corresponding target digital instrument image. The initial digital instrument image is formed by performing an image acquisition operation on the target digital instrument, and the target digital instrument image includes at least the digital display area of ​​the target digital instrument. The first feature mining module is used to perform a first feature mining process on the target digital instrument image to obtain corresponding intermediate image features. The intermediate image features are used to characterize the semantic information of the numbers in the target digital instrument image. The first feature mining process includes at least channel attention processing, which is used to mine the semantic information of the salient channels in the target digital instrument image. The second feature mining module is used to perform second feature mining processing on the intermediate image features to obtain the corresponding target image features. The second feature mining processing is different from the first feature mining processing, and the second feature mining processing includes at least correlation feature mining processing, which is used to mine the semantic information of the correlation between each feature column in the intermediate image features. A semantic recognition module is used to perform semantic recognition processing on the target image features to obtain the target semantic recognition result corresponding to the target digital instrument, wherein the target semantic recognition result includes at least the numbers displayed in the digital display area of ​​the target digital instrument.

9. A digital instrument reading recognition network training device, characterized in that, For implementing the digital instrument reading recognition network training method according to any one of claims 5-7, the digital instrument reading recognition network training device comprises: The network acquisition module is used to acquire the initial image segmentation network and the initial digital instrument reading recognition network included in the initial neural network, and to acquire the first sample digital instrument image, the first image label information corresponding to the first sample digital instrument image, the second sample digital instrument image, and the second image label information corresponding to the second sample digital instrument image, wherein the first image label information is used to indicate the digital display area in the first sample digital instrument image, and the second image label information is used to indicate the number displayed in the digital display area in the second sample digital instrument image; The first network update module is used to segment the first sample digital instrument image using the initial image segmentation network to obtain a corresponding target sample digital instrument image, and to update the network parameters of the initial image segmentation network based on the target sample digital instrument image and the first image label information to obtain a corresponding target image segmentation network, wherein the target sample digital instrument image includes at least the digital display area of ​​the digital instrument. The first sample feature mining module is used to perform a first feature mining process on the second sample digital instrument image using the initial digital instrument reading recognition network to obtain the corresponding sample intermediate image features. The sample intermediate image features are used to characterize the semantic information of the numbers in the second sample digital instrument image, and the first feature mining process includes at least channel attention processing to mine the semantic information of the salient channels in the second sample digital instrument image. The second sample feature mining module is used to perform second feature mining processing on the sample intermediate image features using the initial digital instrument reading recognition network to obtain the corresponding sample target image features. The second feature mining processing is different from the first feature mining processing, and the second feature mining processing includes at least correlation feature mining processing, which is used to mine the semantic information of the correlation between each feature column in the sample intermediate image features. The sample semantic recognition module is used to perform semantic recognition processing on the sample target image features using the initial digital instrument reading recognition network to obtain the sample semantic recognition result corresponding to the digital instrument, wherein the sample semantic recognition result includes at least the numbers displayed in the digital display area of ​​the digital instrument; The second network update module is used to update the network parameters of the initial digital instrument reading recognition network based on the difference between the sample semantic recognition result and the second image label information, so as to obtain the corresponding target digital instrument reading recognition network. The target neural network determination module is used to determine the corresponding target neural network based on the target image segmentation network and the target digital instrument reading recognition network.

10. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor connected to the memory is used to execute the computer program stored in the memory to implement the digital instrument reading recognition method according to any one of claims 1-4, or to implement the digital instrument reading recognition network training method according to any one of claims 5-7.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed, performs the digital instrument reading recognition method according to any one of claims 1-4, or performs the digital instrument reading recognition network training method according to any one of claims 5-7.

Images