Square meter reading identification method and device, program product and electronic equipment

Abstract

The invention belongs to the technical field of instrument intelligent identification, and provides a square meter reading identification method, which comprises the following steps: inputting a collected image and a preset text into a semantic segmentation model, and outputting square meter dial detection frame information and adaptive image transformation parameters; segmenting a square meter dial plate image from the acquired image based on the square meter dial plate detection frame information; performing horizontal correction on the square meter dial image in combination with the self-adaptive image transformation parameters; inputting the horizontally corrected dial plate image of the square meter into a target detection model, wherein the target detection model outputs detection frame information of a pointer, a dial plate, an initial scale and / or a termination scale; and determining the square meter reading according to the pointer, the dial plate and the detection frame information of the initial scale and / or the termination scale. The invention further discloses a square meter reading recognition device, a computer program product and electronic equipment. According to the invention, the accuracy of reading identification can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent meter recognition, and in particular to a square meter reading recognition method, device, program product and electronic equipment. Background Art

[0002] There are a large number of square meters in factories, such as square ammeters, square voltmeters, square pressure gauges, square thermometers, etc. The dial of the square meter usually has a start scale, an end scale, and a pointer. In order to reduce costs, most square meters in factories do not have the remote transmission function of smart meters, and can only be read manually or with the help of external smart devices. In addition, since the factory environment is often in a high temperature or toxic environment, the current safe and scientific way is to use external smart devices for reading, such as using a robot camera to shoot the square meter and perform intelligent reading recognition on the collected image based on deep learning.

[0003] The traditional deep learning scheme directly identifies the dial, starting scale, ending scale and pointer from the robot's captured image. Since the square table is small in size, the image may be blurred when the robot pan / tilt zoom is used to shoot, especially when the pointer is blurred. As a result, even if the traditional deep learning scheme detects the dial, it is difficult to detect the pointer and cannot accurately identify the reading. The reading accuracy of the square table depends largely on the accurate identification of the starting scale, ending scale, pointer, and the relative position relationship between the pointer and the two scales. However, the reading recognition accuracy is closely related to the distortion state of the captured image (especially the horizontal direction of the image plane). In practice, the robot cannot ensure that the square table is on the horizontal line in the captured image. Most of the time, it is tilted. The square table image in the captured image is asymmetrical and distorted, resulting in errors in the relative position relationship between the identified pointer and the two scales. In order to improve the distortion problem, the existing technology performs perspective transformation on the captured image, but the actual effect is limited. After the perspective transformation, there is still distortion in the horizontal direction of the image plane, which affects the accuracy of reading recognition. Summary of the invention

[0004] The present invention aims to at least solve the technical problem of low accuracy of existing square meter reading recognition methods, and provides a square meter reading recognition method, device, program product and electronic equipment.

[0005] In order to achieve the above-mentioned purpose of the present invention, according to the first aspect of the present invention, the present invention provides a square watch reading recognition method, including: acquiring a captured image, inputting the captured image and preset text into a trained semantic segmentation model, the semantic segmentation model outputting square watch dial detection frame information and adaptive image transformation parameters; segmenting a square watch dial image from the captured image based on the square watch dial detection frame information; horizontally correcting the square watch dial image in combination with the adaptive image transformation parameters; inputting the horizontally corrected square watch dial image into a pre-trained target detection model, the target detection model outputting detection frame information of a pointer, a dial, and an initial scale and / or an end scale; determining the square watch reading according to the detection frame information of the pointer, the dial, and the initial scale and / or the end scale.

[0006] Further preferably, the semantic segmentation model includes:

[0007] A text encoder, used to extract text features of a preset text;

[0008] An image encoder, used to extract image features of the captured image;

[0009] The text-image feature fusion module performs cross-modal enhancement processing on text features and image features to obtain enhanced text features and enhanced image features;

[0010] Cross-modal decoder, which obtains decoding features based on enhanced text features and enhanced image features;

[0011] Regression network, which performs regression processing on the decoded features to obtain adaptive image transformation parameters;

[0012] The decoding head obtains the square watch face detection frame information based on the decoding features.

[0013] Further preferably, the semantic segmentation model includes a Grounding DINO network and a regression network, and an input end of the regression network is connected to an output end of a last layer of a fully connected feed-forward network of a decoder of the Grounding DINO network.

[0014] Further preferably, the horizontal correction of the square watch dial image by combining the adaptive image transformation parameters comprises:

[0015] Perform perspective transformation on the square dial image to obtain a perspective transformation matrix, the size of which is 3×3;

[0016] Multiply the first column and the second column of the perspective transformation matrix by the adaptive image transformation parameter to obtain a correction matrix;

[0017] The correction matrix is ​​used to correct the position coordinates of the pixel points in the square watch dial image.

[0018] Further preferably, the determining of the square meter reading according to the pointer, the dial, and the detection frame information of the initial scale and / or the final scale comprises:

[0019] Determine the pointer, the dial, and the initial scale and / or the final scale according to the detection frame information of the pointer, the dial, and the initial scale and / or the final scale;

[0020] If the initial scale is detected and the final scale is not detected, the final scale is obtained by transforming the initial scale symmetrically about the midline of the dial in the image plane;

[0021] If the final scale is detected and the initial scale is not detected, the final scale is transformed symmetrically about the midline of the dial in the image plane to obtain the initial scale;

[0022] Determine the square meter reading based on the pointer, dial, initial scale and final scale.

[0023] Further preferably, the method of determining the square meter reading according to the pointer, the dial, the initial scale and the final scale comprises:

[0024] Connect the initial scale center position and the final scale center position to obtain the first line segment;

[0025] Obtaining the intersection point of the midline of the pointer and the perpendicular bisector of the first line segment, and using the intersection point as the rotation axis point of the pointer;

[0026] Calculate the angle between the line connecting the initial scale center position and the pointer rotation axis point and the pointer midline, and record it as the first angle;

[0027] Calculate the angle between the line connecting the initial scale center position and the pointer rotation axis point and the pointer midline, and record it as the second angle;

[0028] The ratio of the second angle to the first angle is obtained, and the ratio is mapped into a square meter reading using a preset ratio mapping relationship.

[0029] Further preferably, the method of determining the square meter reading according to the pointer, the dial, the initial scale and the final scale includes:

[0030] Connect the initial scale center position and the final scale center position to obtain the first line segment;

[0031] Obtaining the intersection point of the midline of the pointer and the perpendicular bisector of the first line segment, and using the intersection point as the rotation axis point of the pointer;

[0032] Calculate the angle between the line connecting the initial scale center position and the pointer rotation axis point and the pointer midline;

[0033] The included angle is mapped to a square meter reading using a preset angle mapping relationship.

[0034] In order to achieve the above-mentioned object of the present invention, according to a second aspect of the present invention, the present invention provides a square meter reading recognition device, which is used to implement a square meter reading recognition method according to the first aspect of the present invention, comprising:

[0035] The segmentation module obtains the collected image, inputs the collected image and the preset text into the trained semantic segmentation model, and the semantic segmentation model outputs the square watch dial detection frame information and the adaptive image transformation parameters;

[0036] An extraction module, which segments a square watch dial image from a captured image based on the square watch dial detection frame information;

[0037] The correction module combines the adaptive image transformation parameters to perform horizontal correction on the square dial image;

[0038] The target detection module inputs the horizontally corrected square dial image into a pre-trained target detection model, and the target detection model outputs the detection frame information of the pointer, dial, and initial scale and / or terminal scale;

[0039] The reading calculation module determines the square meter reading according to the pointer, the dial, and the detection frame information of the initial scale and / or the final scale.

[0040] In order to achieve the above-mentioned purpose of the present invention, according to the third aspect of the present invention, the present invention provides a computer program product, including a computer program, characterized in that when the computer program is executed by a processor, the steps of a square meter reading identification method described in the first aspect of the present invention are implemented.

[0041] In order to achieve the above-mentioned object of the present invention, according to a fourth aspect of the present invention, the present invention provides an electronic device, the electronic device comprising:

[0042] at least one processor; and,

[0043] A memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform a square meter reading recognition method as described in the first aspect of the present invention.

[0044] The square watch reading recognition scheme provided by the present invention divides the reading recognition process into two stages. In the first stage, the semantic segmentation model is used to obtain the square watch dial detection frame information and adaptive image transformation parameters of the collected image, and the square watch dial image is segmented and extracted from the collected image according to the square watch dial detection frame information; in the second stage, the square watch dial image is horizontally corrected in combination with the adaptive image transformation parameters to improve the horizontality of the corrected square watch dial image, and then the corrected square watch dial image is input into the target detection model to obtain the pointer, dial, and the initial scale and / or the end scale detection frame information, and finally the square watch reading is obtained based on the pointer, dial, and the initial scale and / or the end scale detection frame information, which has the following beneficial technical effects:

[0045] (1) The first stage combines preset text semantic information to detect square tables, which can achieve high-precision semantic segmentation of square tables on low-quality image acquisition screens, effectively solving the problem of low accuracy or even difficulty in reading of traditional algorithms. The subsequent second-stage target detection model only processes the square table dial image. This staged processing method can reduce the target detection range, improve detection performance, and avoid a large number of missed detections and false detections.

[0046] (2) In the first stage, adaptive image transformation parameters were obtained and the traditional perspective transformation matrix was improved, which effectively solved the image distortion phenomenon and improved the horizontality of the corrected square dial image, thereby improving the accuracy of the subsequent reading recognition step. In the subsequent reading recognition step, the missing starting scale or ending scale can be inferred based on the symmetry of the starting scale and the ending scale, which effectively solved the problem of blurred pointers and inability to read when the initial scale or the ending scale is small in the square dial image. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a schematic flow chart of a square meter reading recognition method in a preferred embodiment of the present invention;

[0048] Figure 2 is a structural schematic diagram of a semantic segmentation model in a preferred embodiment of the present invention;

[0049] Figure 3 is a schematic diagram of a captured image in an example of the present invention;

[0050] Figure 4 is based on Figure 2 Extracted square watch face image;

[0051] Figure 5 yes Figure 3 Schematic diagram of the pointer detection box output from the target detection image;

[0052] Figure 6It is a schematic diagram of the whole process of square meter reading recognition in an application scenario of the present invention;

[0053] Figure 7 It is a schematic structural diagram of an electronic device in a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0054] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0055] In the description of the present invention, it is necessary to understand that the terms "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0056] In the description of the present invention, unless otherwise specified and limited, it should be noted that the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a mechanical connection or an electrical connection, or it can be the internal connection between two components. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to the specific circumstances.

[0057] The execution subject of the square meter reading recognition method includes but is not limited to at least one of the electronic devices that can be configured to execute the method provided in the embodiment of the present application, such as a server, a terminal, an inspection robot, and an inspection robot management platform. In other words, the square meter reading recognition method can be executed by software or hardware installed in a terminal device or a server device or an inspection robot or an inspection robot management platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc. The server can be an independent server, or it can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms.

[0058] The present invention discloses a method for identifying square meter readings. In a preferred embodiment, Figure 1As shown, the method includes:

[0059] Step S1, obtaining a captured image, inputting the captured image and preset text into a trained semantic segmentation model, and the semantic segmentation model outputs square watch dial detection frame information and adaptive image transformation parameters.

[0060] In this embodiment, the captured image may be an image taken by a patrol robot in the factory, or may be a fixed camera near the square meter for photographing the square meter. The square meter is preferably, but not limited to, a square ammeter, a square voltmeter, a square pressure meter, or a square temperature meter. Figure 3 In the example shown, it can be seen that the square watch dial image in the captured image only occupies a small part of the entire image. The preset text is used to provide semantic information for identifying the square watch. Specifically, the preset text is preferably but not limited to "identify the square watch" or "identify the square watch dial" or "identify the square watch dial", which can help the semantic segmentation model accurately identify the square watch dial image. The square watch dial detection frame information can specifically be the position coordinates of the upper left corner and lower right corner of the square watch dial detection frame in the captured image on the image plane.

[0061] Step S2, segmenting a square watch dial image from the captured image based on the square watch dial detection frame information. Figure 4 The square watch dial image segmented from the acquired image in the above example is shown.

[0062] Step S3, horizontally correcting the square watch dial image in combination with the adaptive image transformation parameters.

[0063] Preferably, step S3 specifically includes:

[0064] Step S31, perform perspective transformation on the square dial image to obtain a perspective transformation matrix. The perspective transformation matrix has a size of 3×3 and is in the form of:

[0065] Step S312: transform the first column of the perspective transformation matrix and the second column Multiply by the adaptive image transformation parameter α to obtain the correction matrix

[0066] Step S33, using the correction matrix to correct the position coordinates of the pixel points in the square dial image:

[0067]

[0068] Among them, (x, y, 1) represents the coordinates of the pixel position in the square watch dial image before correction, and (X, Y, Z) represents the coordinates of the pixel position in the square watch dial image after correction.

[0069] Step S4, inputting the horizontally corrected square dial image into a pre-trained target detection model, and the target detection model outputs the detection frame information of the pointer, dial, and initial scale and / or terminal scale.

[0070] In this embodiment, the target detection model is a multi-target detection model, and its network structure is preferably but not limited to a YOLO series network, such as YOLO-World. Due to the influence of shooting angle, shooting stability, etc., the initial scale or the final scale may be lost in the square dial image. Therefore, the target detection model outputs the detection result, including the detection frame information of the pointer, dial and initial scale, or the detection frame information of the pointer, dial and final scale, or the detection frame information of the pointer, dial, initial scale and final scale. Figure 5 It shows that in the above example, the detection box information output by the target detection model is the detection box position information, such as the pointer detection box information is bbox={x1,y1,x2,y2}, (x1,y1) represents the position coordinates of the upper left corner of the pointer detection box on the image plane, and (x2,y2) represents the position coordinates of the lower right corner of the pointer detection box on the image plane.

[0071] Step S5, determining the square meter reading according to the pointer, the dial, and the detection frame information of the initial scale and / or the final scale.

[0072] In a preferred embodiment, Figure 2 As shown in the figure, in order to improve the horizontal correction effect and use the semantic information of the preset text to extract the square table detection accuracy, the semantic segmentation model includes:

[0073] A text encoder is used to extract text features of a preset text. The text encoder is preferably, but not limited to, a BERT encoder or a CLIP model (Contrastive Language-Image Pre-Training, a multimodal pre-trained neural network). The CLIP model generates text features based on the preset text.

[0074] The image encoder is used to extract image features of the captured image. The image encoder is preferably, but not limited to, SwinTransformer, which can extract multi-scale image features.

[0075] The text-image feature fusion module performs cross-modal enhancement processing on text features and image features to obtain enhanced text features and enhanced image features. Specifically, the image features can be enhanced using a deformable self-attention mechanism, and the text features can be enhanced using a common self-attention mechanism.

[0076] The cross-modal decoder obtains decoding features based on enhanced text features and enhanced image features.

[0077] The regression network performs regression processing on the decoded features to obtain adaptive image transformation parameters; the regression network can be a single fully connected layer or two or more cascaded fully connected layers.

[0078] The decoding head obtains the detection frame information of the square watch dial based on the decoding features. The decoding head is an existing detection frame acquisition module, which can be a Region Proposal Network (RPN) or a target frame query mechanism (such as Query) module.

[0079] In this embodiment, the decoded features of the acquired image learned by the semantic segmentation model are used to adaptively generate adaptive image transformation parameters to optimize the subsequent image correction matrix.

[0080] In this embodiment, preferably, in order to simplify the network structure, the semantic segmentation model may include an existing semantic segmentation network and a regression network, and the regression network is connected to the previous layer network of the detection frame acquisition unit of the semantic segmentation network. The existing semantic segmentation network is preferably, but not limited to, mask-rcnn, openseed or Grounding DINO.

[0081] In this embodiment, in order to make full use of the unsupervised characteristics of Grounding DINO, it is further preferred that the semantic segmentation model includes a Grounding DINO network and a regression network, and the input end of the regression network is connected to the output end of the last layer of the fully connected feedforward network (FFN) of the decoder of the Grounding DINO network. This can better adapt to the square watch dials with different shapes in the factory, and realize accurate semantic segmentation of various square watches. It is also suitable for small batch training samples of factory scenes, so that the segmentation accuracy of the semantic segmentation model does not depend on the number of samples. At this time, the adaptive image transformation parameter α obtained is expressed as:

[0082] α=f(x,θ)+∈

[0083] Among them, f(x,θ) indicates that the regression network uses its network parameters θ and the decoded features x output by the last layer of the fully connected feedforward network (FFN) of the decoder of the Grounding DINO network to obtain the adaptive image transformation parameter α, and introduces an error ∈ that conforms to the normal distribution to improve robustness.

[0084] In this embodiment, the semantic segmentation model and the object detection model can be trained independently and in parallel to improve training efficiency.

[0085] In this embodiment, the training process of the semantic segmentation model includes:

[0086] Step A1, construct a semantic segmentation sample image set, collect more than one square table collection image collected from the factory history (such as Figure 3) as a semantic segmentation sample image, manually mark the detection frame label of the square watch dial area in each square watch acquisition image, obtain the detection frame information (i.e., position information), and calculate the true value of the adaptive image transformation parameter;

[0087] Step A2, constructing the network structure of the semantic segmentation model and initializing the network parameters; dividing the semantic segmentation sample image set into a training set, a test set, and a validation set;

[0088] Step A3, training the semantic segmentation model using the training set, calculating the semantic segmentation loss function during the training process, and updating the network parameters of the semantic segmentation model using the gradient back propagation algorithm according to the semantic segmentation loss function until the semantic segmentation training stop condition is reached;

[0089] Step A4, use the test set and the validation set to test and verify the trained semantic segmentation model respectively. If the test and validation pass, save the network parameters to obtain the trained semantic segmentation model. If the test and validation fail, return to execute steps A3 and A4.

[0090] In this embodiment, the semantic segmentation training stop condition is preferably, but not limited to, that the number of semantic segmentation training times reaches a preset maximum number of training times, or that the semantic segmentation loss function value is less than or equal to a preset loss threshold. The semantic segmentation loss function L1 is:

[0091]

[0092] Among them, IOU i represents the intersection-over-union ratio of the detection box obtained by the semantic segmentation model in the training of the i-th semantic segmentation sample image input and the detection box label of the i-th semantic segmentation sample image; α i represents the adaptive image transformation parameter obtained by the semantic segmentation model in the training of the i-th semantic segmentation sample image input, α′ i Represents the true value of the adaptive image transformation parameter of the i-th semantic segmentation sample image. N1 represents the number of semantic segmentation sample images involved in the training.

[0093] In this embodiment, the method for obtaining the true value of the adaptive image transformation parameter of the semantic segmentation sample image is:

[0094] Step B1, segmenting a square watch dial image from the semantic segmentation sample image using the detection frame label of the semantic segmentation sample image;

[0095] Step B2, performing perspective transformation on the square watch dial image to obtain a perspective transformation matrix;

[0096] Step B3, within the value range (0, 1] of the adaptive image transformation parameter, the value of the adaptive image transformation parameter is continuously increased according to a preset step size (such as a step size of 0.01 or 0.05 or 0.1) within the value range (0, 1), and after each stepping by one step, the corrected square watch dial image is obtained according to the above transformation formula, and multiple experts perform non-distortion scoring on the corrected square watch dial image. After removing the largest non-distortion score and the smallest non-distortion score, the average of the remaining non-distortion scores is calculated and the average is used as the average score. If the average score is greater than the preset score threshold, the current adaptive image transformation parameter value is used as the adaptive image transformation parameter true value of the semantic segmentation sample image. If the average score is less than or equal to the preset score threshold, the adaptive image transformation parameter value is increased by one step, and the next step is judged. If the adaptive image transformation parameter true value of the semantic segmentation sample image is still not found after stepping through the entire value range (0, 1), the step size is changed and step B3 is continued. Preferably, the step size is further reduced.

[0097] In this embodiment, the training process of the target detection model includes:

[0098] Step C1, construct a target detection sample image set. Collect more than one square table collection image of the factory history collection (such as Figure 3 ), in each square watch acquisition image, a square watch dial image is manually extracted by using a rectangular frame segmentation and used as a target detection sample image, and in each target detection sample image, a pointer, a dial, a starting scale and / or an ending scale are manually marked by using a rectangular frame;

[0099] Step C2, constructing the network structure of the target detection model and initializing the network parameters; dividing the target detection sample image set into a training set, a test set and a validation set;

[0100] Step C3, training the target detection model using the training set, calculating the target detection loss function during the training process, and updating the network parameters of the target detection model using the gradient back propagation algorithm according to the target detection loss function until the target detection training stop condition is reached;

[0101] Step C4, use the test set and the validation set to test and verify the trained target detection model respectively. If the test and validation pass, save the network parameters to obtain the trained target detection model. If the test and validation fail, return to execute steps C3 and C4.

[0102] In this embodiment, the target detection training stop condition is preferably, but not limited to, that the target detection training times reaches a preset maximum training times, or the target detection loss function is less than or equal to a preset loss threshold. The target detection loss function L2 is:

[0103]

[0104] Where N2 represents the number of target detection sample images participating in the training. M represents the number of targets in the jth target detection sample image. For example, if the target detection sample image includes four targets: pointer, dial, start scale, and end scale, then M is 4, and m is the target index in the jth target detection sample image. j,m It represents the intersection-over-union ratio of the mth target in the jth target detection sample image.

[0105] In a preferred embodiment, in step S5, determining the square meter reading according to the pointer, the dial, and the detection frame information of the initial scale and / or the final scale includes:

[0106] Step S51, the pointer, dial, and initial scale and / or final scale are determined according to the detection frame information of the pointer, dial, and initial scale and / or final scale. Specifically, the positions of the pointer / dial / initial scale / final scale can be determined according to the detection frame information of the pointer / dial / initial scale / final scale, and then the image area of ​​the pointer / dial / initial scale / final scale is obtained.

[0107] Step S52: If the initial scale is detected and the terminal scale is not detected, the initial scale is symmetrically transformed about the center line of the dial (i.e., the center axis of the corrected square dial image) in the image plane to obtain the terminal scale; the initial scale and the terminal scale are symmetrical about the center line of the dial (i.e., the center axis), and the symmetrical relationship is used to obtain a symmetrical terminal scale, so that the reading can be recognized even when the terminal scale is missing.

[0108] If the terminal scale is detected and the initial scale is not detected, the terminal scale is transformed symmetrically about the center line of the dial in the image plane to obtain the initial scale; the initial scale and the terminal scale are symmetrical about the center line of the dial (i.e., the center axis of the corrected square dial image), and this symmetry relationship is used to obtain a symmetrical starting scale, so that the reading can be recognized when the starting scale is missing.

[0109] Determine the square meter reading based on the pointer, dial, initial scale and final scale.

[0110] In this embodiment, when the initial scale or the final scale is missing, another scale is used to infer the missing scale. The accuracy of reading recognition is effectively improved by the post-processing algorithm of inferring the initial scale or the final scale.

[0111] In a preferred solution of this embodiment, in step S52, determining the square meter reading according to the pointer, the dial, the initial scale and the final scale includes:

[0112] Connect the initial scale center position and the final scale center position to obtain the first line segment;

[0113] Obtain the intersection point of the straight line where the pointer is located and the perpendicular bisector of the first line segment (the perpendicular bisector is both the midline and perpendicular line of the first line segment), and use the intersection point as the rotation axis point of the pointer;

[0114] Calculate the angle between the line connecting the initial scale center position and the pointer rotation axis point and the pointer center line, which is recorded as the first angle; specifically, draw a line connecting the initial scale center position and the pointer rotation axis point, which is recorded as the first line; draw a line connecting the terminal scale center position and the pointer rotation axis point, which is recorded as the second line, and record the angle (acute angle) between the first line and the second line as the first angle.

[0115] Calculate the angle between the line connecting the initial scale center position and the pointer rotation axis point and the pointer midline, and record it as the second angle;

[0116] The ratio of the second angle to the first angle is obtained, and the ratio is mapped to a square table reading using a preset ratio mapping relationship. The ratio mapping relationship is preferably, but not limited to, a linear equation obtained by linear fitting of multiple sets of ratios-readings, or a lookup table formed by multiple sets of ratios-readings, or an interpolation conversion based on multiple sets of ratios-readings.

[0117] In another preferred embodiment of the present embodiment, in step S52, determining the square meter reading according to the pointer, the dial, the initial scale and the final scale includes:

[0118] Determine the pointer, the dial, the initial scale and the final scale according to the detection frame information of the pointer, the dial, the initial scale and the final scale;

[0119] Connect the initial scale center position and the final scale center position to obtain the first line segment;

[0120] Obtaining the intersection point of the midline of the pointer and the perpendicular bisector of the first line segment, and using the intersection point as the rotation axis point of the pointer;

[0121] Calculate the angle between the line connecting the initial scale center position and the pointer rotation axis point and the pointer midline;

[0122] The angle is mapped to a square table reading using a preset angle mapping relationship. The angle mapping relationship is preferably, but not limited to, a linear equation obtained by linear fitting of multiple sets of angle readings, or a lookup table composed of multiple sets of angle readings, or an interpolation conversion based on multiple sets of angle readings.

[0123] Figure 6It is a schematic diagram of the entire process of square meter reading recognition in an application scenario of the present invention. In this application scenario, after the reading is obtained, the pointer is mapped back to the square meter dial image before correction and compared with the pointer in the square meter dial image before correction for verification.

[0124] The present invention also discloses a square meter reading recognition device, which is used to implement the square meter reading recognition method provided by the present invention. In a preferred embodiment, the device comprises:

[0125] The segmentation module obtains the collected image, inputs the collected image and the preset text into the trained semantic segmentation model, and the semantic segmentation model outputs the square watch dial detection frame information and the adaptive image transformation parameters;

[0126] An extraction module, which segments a square watch dial image from a captured image based on the square watch dial detection frame information;

[0127] The correction module combines the adaptive image transformation parameters to perform horizontal correction on the square dial image;

[0128] The target detection module inputs the horizontally corrected square dial image into a pre-trained target detection model, and the target detection model outputs the detection frame information of the pointer, dial, and initial scale and / or terminal scale;

[0129] The reading calculation module determines the square meter reading according to the pointer, the dial, and the detection frame information of the initial scale and / or the final scale.

[0130] In this embodiment, the segmentation module, extraction module, correction module, target detection module and reading calculation module correspond one by one to step S1, step S2, step S3, step S4 and step S5 of the above-mentioned square table reading recognition method, and will not be repeated here.

[0131] The present invention also discloses a computer program product, including a computer program, which, when executed by a processor, implements the steps of the above-mentioned square meter reading identification method provided by the present invention. The computer program product should be understood as a software product that mainly implements its solution through a computer program, such as a program product integrated in the cloud or a software library.

[0132] The present invention also discloses an electronic device. In one embodiment, the electronic device includes at least one processor; and a memory connected to the at least one processor in communication; wherein:

[0133] The memory stores a computer program executable by the at least one processor. The computer program is executed by the at least one processor so that the at least one processor can execute the square meter reading identification method provided by the present invention.

[0134] like Figure 7 , which is a schematic diagram of the structure of an electronic device for a square meter reading recognition method provided by an embodiment of the present invention. The electronic device may include a processor 10, a memory 11, a communication bus 12, and a communication interface 13, and may also include a computer program stored in the memory 11 and executable on the processor 10, such as a square meter reading recognition method program.

[0135] The processor 10 may be composed of an integrated circuit in some embodiments, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips, etc. The processor 10 is the control core (Control Unit) of the electronic device, and uses various interfaces and lines to connect various components of the entire electronic device, and executes or executes programs or modules stored in the memory 11 (for example, executing a square meter reading recognition method, etc.), and calls data stored in the memory 11 to execute various functions of the electronic device and process data.

[0136] The memory 11 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a mobile hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 11 may be an internal storage unit of an electronic device, such as a mobile hard disk of the electronic device. In other embodiments, the memory 11 may also be an external storage device of the electronic device, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Further, the memory 11 may also include both an internal storage unit of the electronic device and an external storage device. The memory 11 may not only be used to store application software and various types of data installed in the electronic device, such as the code of the square meter reading recognition method program, etc., but also be used to temporarily store data that has been output or is to be output.

[0137] The communication bus 12 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is configured to realize connection and communication between the memory 11 and at least one processor 10, etc.

[0138] The communication interface 13 is used for communication between the above-mentioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device and other electronic devices. The user interface may be a display (Display), an input unit (such as a keyboard (Keyboard)), and optionally, the user interface may also be a standard wired interface, a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode, organic light-emitting diode) touch device, etc. Among them, the display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device and to display a visual user interface.

[0139] Figure 7 Only an electronic device with components is shown, and those skilled in the art will understand that Figure 7 The structure shown does not constitute a limitation on the electronic device, and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0140] For example, although not shown, the electronic device may also include a power source (such as a battery) for supplying power to various components. Preferably, the power source may be logically connected to at least one processor 10 through a power management device, so that the power management device can realize functions such as charging management, discharging management, and power consumption management. The power source may also include any components such as one or more DC or AC power sources, recharging devices, power failure detection circuits, power converters or inverters, and power status indicators. The electronic device may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0141] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited by this structure.

[0142] Furthermore, if the module / unit integrated in the electronic device 1 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).

[0143] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", "an implementation", "a preferred implementation" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0144] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. A method for identifying square meter readings, characterized in that: include: Obtain a captured image, input the captured image and preset text into a trained semantic segmentation model, and the semantic segmentation model outputs square watch dial detection frame information and adaptive image transformation parameters; Segmenting a square watch dial image from the captured image based on the square watch dial detection frame information; The square dial image is horizontally corrected by combining adaptive image transformation parameters; The horizontally corrected square dial image is input into a pre-trained object detection model, and the object detection model outputs the detection frame information of the pointer, dial, and initial scale and / or terminal scale; The square meter reading is determined based on the pointer, dial, and detection frame information of the initial scale and / or the final scale.

2. A square meter reading recognition method as claimed in claim 1, characterized in that: The semantic segmentation model includes: A text encoder, used to extract text features of a preset text; An image encoder, used to extract image features of the captured image; The text-image feature fusion module performs cross-modal enhancement processing on text features and image features to obtain enhanced text features and enhanced image features; Cross-modal decoder, which obtains decoding features based on enhanced text features and enhanced image features; Regression network, which performs regression processing on the decoded features to obtain adaptive image transformation parameters; The decoding head obtains the square watch face detection frame information based on the decoding features.

3. A square meter reading recognition method as claimed in claim 1, characterized in that: The semantic segmentation model includes a Grounding DINO network and a regression network, wherein an input end of the regression network is connected to an output end of a last layer of a fully connected feedforward network of a decoder of the Grounding DINO network.

4. A square meter reading recognition method as claimed in claim 1, characterized in that: The method of horizontally correcting the square watch dial image by combining the adaptive image transformation parameters includes: Perform perspective transformation on the square dial image to obtain a perspective transformation matrix, the size of which is 3×3; Multiply the first column and the second column of the perspective transformation matrix by the adaptive image transformation parameter to obtain a correction matrix; The correction matrix is ​​used to correct the position coordinates of the pixel points in the square watch dial image.

5. A square meter reading recognition method as claimed in any one of claims 1 to 4, characterized in that: The method of determining the square meter reading according to the pointer, the dial, and the detection frame information of the initial scale and / or the final scale includes: Determine the pointer, the dial, and the initial scale and / or the final scale according to the detection frame information of the pointer, the dial, and the initial scale and / or the final scale; If the initial scale is detected and the final scale is not detected, the final scale is obtained by transforming the initial scale symmetrically about the midline of the dial in the image plane; If the final scale is detected and the initial scale is not detected, the final scale is transformed symmetrically about the midline of the dial in the image plane to obtain the initial scale; Determine the square meter reading based on the pointer, dial, initial scale and final scale.

6. A square meter reading recognition method as claimed in claim 5, characterized in that: The method of determining the square meter reading according to the pointer, the dial, the initial scale and the final scale includes: Connect the initial scale center position and the final scale center position to obtain the first line segment; Obtaining the intersection point of the midline of the pointer and the perpendicular bisector of the first line segment, and using the intersection point as the rotation axis point of the pointer; Calculate the angle between the line connecting the initial scale center position and the pointer rotation axis point and the pointer midline, and record it as the first angle; Calculate the angle between the line connecting the initial scale center position and the pointer rotation axis point and the pointer midline, and record it as the second angle; The ratio of the second angle to the first angle is obtained, and the ratio is mapped into a square meter reading using a preset ratio mapping relationship.

7. A square meter reading recognition method as claimed in claim 5, characterized in that: The method of determining the square meter reading according to the pointer, the dial, the initial scale and the final scale includes: Connect the initial scale center position and the final scale center position to obtain the first line segment; Obtaining the intersection point of the midline of the pointer and the perpendicular bisector of the first line segment, and using the intersection point as the rotation axis point of the pointer; Calculate the angle between the line connecting the initial scale center position and the pointer rotation axis point and the pointer midline; The included angle is mapped to a square meter reading using a preset angle mapping relationship.

8. A square meter reading recognition device, characterized in that: A square meter reading recognition method for implementing any one of claims 1 to 7, comprising: The segmentation module obtains the collected image, inputs the collected image and the preset text into the trained semantic segmentation model, and the semantic segmentation model outputs the square watch dial detection frame information and the adaptive image transformation parameters; An extraction module, which segments a square watch dial image from a captured image based on the square watch dial detection frame information; The correction module combines the adaptive image transformation parameters to perform horizontal correction on the square dial image; The target detection module inputs the horizontally corrected square dial image into a pre-trained target detection model, and the target detection model outputs the detection frame information of the pointer, dial, and initial scale and / or terminal scale; The reading calculation module determines the square meter reading according to the pointer, the dial, and the detection frame information of the initial scale and / or the final scale.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of a square meter reading recognition method described in any one of claims 1-7 are implemented.

10. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, A memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor so that the at least one processor can execute a square meter reading recognition method as described in any one of claims 1 to 7.

Images