Scale value detection method and device, equipment, storage medium and product

Abstract

The invention discloses a scale value detection method and device, equipment, a storage medium and a product, and the method comprises the steps: carrying out the processing of a to-be-detected image through a pre-trained image processing model, and obtaining a segmentation result; the segmentation result comprises a scale and marker post segmentation result and a first scale unit sequence segmentation result; according to the distance between the adjacent scale units, checking and correcting the sequence segmentation result of the first scale unit to obtain a sequence segmentation result of a second scale unit; and obtaining a target scale value of the marker post indicating the scale in the to-be-detected image according to the scale and marker post segmentation result and the second scale unit sequence segmentation result. According to the invention, the accuracy and precision of scale value detection can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target detection, and in particular to a scale value detection method and device, equipment, a storage medium and a product. BACKGROUND

[0002] The existing engineering construction quality inspection, especially size quality inspection, mainly measures by hand and fills the value into the system, which is low in efficiency. With the development of AI technology, AI technology can assist the engineering construction quality inspection to improve the efficiency. For example, for a communication operator, in the engineering construction scene of laying optical cable and building optical box pedestal, some trenches and pits need to be excavated, and the stability of the optical box will be affected after the non-standard size pedestal is solidified and formed. Therefore, a size measurement scheme of "ruler (tape measure) + pole" is designed, which requires the construction personnel to measure and record by using the ruler. In order to facilitate the detection of the scale value indicated by the AI technology, the construction personnel are required to place a "pole" horizontally on the ground and ensure that the pole and the ruler are in the intersecting state. The above-mentioned "ruler + pole" method lays a foundation for detecting the size of hidden engineering (trench, pit, optical box pedestal, etc.) through images.

[0003] The existing scale value detection method mainly relates to the detection of the scale of a water gauge, which is suitable for simple scenes with vertical water gauges, clear water line marks and single scale types. However, this scheme is difficult to adapt to complex engineering construction scenes. The photographed images are prone to scale detection missing due to factors such as occlusion, shooting angle and image blur, which causes false detection of scale values and makes it difficult to meet the accuracy requirements. SUMMARY

[0004] The present application provides a scale value detection method, device, equipment, storage medium and product to solve the problem that the existing technology does not have the precision requirement of detecting the scale value due to the factors such as occlusion, shooting angle and image blur in the photographed images.

[0005] To achieve the above-mentioned purpose, the present application provides a scale value detection method, which comprises the following steps:

[0006] Obtaining a to-be-detected image;

[0007] Processing the to-be-detected image through a pre-trained image processing model to obtain a segmentation result; the segmentation result comprises a ruler and pole segmentation result and a first scale unit sequence segmentation result; the image processing model is obtained by multi-task training using an image sample set, the multi-task at least comprises a first task and a second task, the first task is used to extract the ruler and pole from the image, the second task is used to extract the first scale unit sequence from the image, the first scale unit sequence comprises at least one scale unit, and each scale unit is composed of multiple scale values.

[0008] According to the distance between adjacent scale units, the first scale unit sequence segmentation result is checked and corrected to obtain a second scale unit sequence segmentation result;

[0009] According to the ruler and rod segmentation result and the second scale unit sequence segmentation result, a target scale value of a rod indicating a ruler in the to-be-detected image is obtained.

[0010] As an improvement of the above scheme, the target scale value of the rod indicating the ruler in the to-be-detected image is obtained according to the ruler and rod segmentation result and the second scale unit sequence segmentation result, comprising:

[0011] According to the ruler and rod segmentation result, a position of an intersection of the ruler and the rod in the to-be-detected image is obtained;

[0012] According to the intersection position, a candidate scale unit in which the target scale value falls in the second scale unit sequence segmentation result is determined;

[0013] According to the position of the candidate scale unit, the target scale value is obtained.

[0014] As an improvement of the above scheme, the target scale value is composed of a target primary scale value and a target secondary scale value;

[0015] The target scale value is obtained according to the position of the candidate scale unit, comprising:

[0016] According to the segmentation result, the target primary scale value is obtained;

[0017] According to the position of the candidate scale unit and the intersection position, a length of the candidate scale unit and a length difference between the position of the candidate scale unit and the intersection position are obtained;

[0018] A proportion of the length difference in the length is calculated, and the target secondary scale value is obtained according to the proportion.

[0019] As an improvement of the above scheme, the target primary scale value is obtained according to the segmentation result, comprising:

[0020] If a scale starting point of a ruler is recognized in the segmentation result, a number of scale units before the intersection position is obtained according to the second scale unit sequence segmentation result, and the target primary scale value is obtained according to the number.

[0021] As an improvement of the above scheme, the first scale unit sequence segmentation result is checked and corrected according to the distance between adjacent scale units to obtain a second scale unit sequence segmentation result, comprising:

[0022] According to the distance between adjacent scale units, whether there is a missed scale unit in the first scale unit sequence segmentation result is verified;

[0023] If not, the first scale unit sequence segmentation result is taken as the second scale unit sequence segmentation result;

[0024] If so, adjacent abnormal scale units before and after the missed scale unit are determined; according to the positions of the adjacent abnormal scale units, the position of the missed scale unit is obtained; and according to the position of the missed scale unit, the first scale unit sequence segmentation result is corrected to obtain the second scale unit sequence segmentation result.

[0025] As an improvement of the above scheme, the verification of whether there is a missed scale unit in the first scale unit sequence segmentation result according to the distance between adjacent scale units comprises:

[0026] The first center point position of each scale unit in the first scale unit sequence segmentation result is calculated;

[0027] According to each first center point position, all scale units in the first scale unit sequence segmentation result are reordered;

[0028] The reordered first center point positions are fitted by the least square method to obtain a center point straight line;

[0029] In the direction of the center point straight line, the distance between adjacent reordered first center point positions is calculated to determine whether there is a distance meeting a preset condition.

[0030] As an improvement of the above scheme, the position of the missed scale unit is obtained according to the positions of the adjacent abnormal scale units before and after the missed scale unit, which comprises:

[0031] According to the positions of the adjacent abnormal scale units before and after the missed scale unit, a second center point position of the missed scale unit is obtained;

[0032] According to the second center point position, the position of the missed scale unit is obtained.

[0033] As an improvement of the above scheme, the loss function of the image processing model is composed of at least a first loss function of the first task and the second task, and the first loss function is used to measure the difference between the model segmentation result and the real label.

[0034] As an improvement of the above-mentioned scheme, the multi-task further comprises: a third task for detecting a main scale value of a staff indicated by a staff pointer in an image, the loss function of the image processing model further comprises a second loss function of the third task, and the second loss function comprises a first sub-loss function, a second sub-loss function and a third sub-loss function; the first sub-loss function is used to measure the difference between a main scale value predicted category and a main scale value real category, the second sub-loss function is used to determine whether a target detection frame includes a main scale value, and the fourth loss function is used to measure the difference between a main scale value detection frame and a main scale value annotation frame.

[0035] As an improvement of the above-mentioned scheme, the image processing model comprises one shared encoder and at least two decoders; the at least two decoders comprise a first decoder and a second decoder, the first decoder is used to perform the first task, and the second decoder is used to perform the second task.

[0036] To achieve the above-mentioned purpose, the embodiment of the present application further provides a scale value detection device, comprising:

[0037] An acquisition module is configured to acquire a to-be-detected image.

[0038] A processing module is configured to process the to-be-detected image by using a pre-trained image processing model to obtain a segmentation result; the segmentation result comprises a staff and staff pointer segmentation result and a first scale unit sequence segmentation result; the image processing model is obtained by using an image sample set to perform multi-task training, the multi-task at least comprises a first task and a second task, the first task is used to extract a staff and staff pointer from an image, and the second task is used to extract a first scale unit sequence from an image; the first scale unit sequence comprises at least one scale unit, and each scale unit comprises a plurality of scale values.

[0039] A correction module is configured to verify and correct the first scale unit sequence segmentation result according to the distance between adjacent scale units to obtain a second scale unit sequence segmentation result.

[0040] A detection module is configured to obtain a target scale value of a staff pointer indicating a staff in the to-be-detected image according to the staff and staff pointer segmentation result and the second scale unit sequence segmentation result.

[0041] To achieve the above-mentioned purpose, the embodiment of the present application further provides a scale value detection device, comprising a processor, a memory and a computer program stored in the memory and configured to be executed by the processor; when the processor executes the computer program, the scale value detection method as described above is realized.

[0042] To achieve the above object, the embodiment of the present application further provides a computer readable storage medium, which comprises a stored computer program; wherein the computer program controls the device where the computer readable storage medium is located to execute the scale value detection method as described above when running.

[0043] To achieve the above object, the embodiment of the present application further provides a computer program product, which comprises computer program / instruction, and the computer program / instruction is executed by a processor to realize the scale value detection method as described above.

[0044] Compared with the prior art, the scale value detection method, device, equipment, storage medium and product provided by the embodiment of the present application can effectively avoid the problem of low detection precision caused by too small scale value by taking multiple scale values as a scale unit for image processing, and improve the recognition detection precision; by checking and correcting the first scale unit sequence segmentation result, the scale missing detection phenomenon caused by factors such as shielding, shooting angle, image blur and the like can be avoided, and the accuracy and precision of scale value detection are improved. The embodiment of the present application has strong robustness, and has high precision for scale value detection under various complex conditions such as different backgrounds, different types of scales and poles, different shooting distances and angles, and is suitable for various complex engineering construction scenes. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 is a flowchart of a scale value detection method provided by the embodiment of the present application;

[0046] Figure 2 is an example diagram of a scale unit provided by the embodiment of the present application;

[0047] Figure 3 is a structure diagram of an image processing model provided by the embodiment of the present application;

[0048] Figure 4 is a structure block diagram of a scale value detection device provided by the embodiment of the present application;

[0049] Figure 5 is a structure block diagram of a scale value detection device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0050] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0051] Referring toFigure 1 , Figure 1 is a flowchart of a scale value detection method provided by an embodiment of the present application. The target scale value detection method comprises:

[0052] S1, obtaining a to-be-detected image;

[0053] S2, processing the to-be-detected image by using a pre-trained image processing model to obtain a segmentation result; the segmentation result comprises a ruler and staff segmentation result and a first scale unit sequence segmentation result; the image processing model is obtained by using an image sample set for multi-task training, the multi-task at least comprises a first task and a second task, the first task is used for extracting a ruler and staff from an image, the second task is used for extracting a first scale unit sequence from an image, the first scale unit sequence comprises at least one scale unit, and each scale unit is composed of multiple scale values;

[0054] S3, according to the distance between adjacent scale units, the first scale unit sequence segmentation result is checked and corrected to obtain a second scale unit sequence segmentation result;

[0055] S4, according to the ruler and staff segmentation result and the second scale unit sequence segmentation result, a target scale value of a staff indicating a ruler in the to-be-detected image is obtained.

[0056] It is worth noting that in an image, the scale usually accounts for a very small proportion, which belongs to small target detection and is a relatively difficult task. In order to reduce the interference of other regions in the image as much as possible and ensure the accuracy of subsequent detection, the present application embodiment positions and segments the ruler and staff, and positions and segments the scale unit sequence by taking multiple scale values as a whole, i.e., a scale unit, so that the target is easier to detect, effectively avoids the problem of low detection precision caused by too small scale values, and improves the recognition detection precision. Figure 2 The scale value can be regarded as a scale unit in units of ten centimeters, i.e., the scale unit unit is 10, which can be used for subsequent calculation, Figure 2 (a) and Figure 2 (b) are two cases of actual scale units in the to-be-detected image, and there are other actual cases.

[0057] In addition, due to the problems of blur, deformation, small target, etc. of the ruler in the actual scene, the first scale unit sequence segmentation result often has the problem of missed detection. Based on the integrity of the ruler and the continuity of the scale, the present application embodiment checks and corrects the first scale unit sequence segmentation result by using the distance between adjacent scale units, thereby improving the stability and precision of detection.

[0058] Specifically, the ruler and pole segmentation result includes: a ruler and pole image obtained by segmentation processing, a ruler position and a pole position; and the first scale unit sequence segmentation result includes: a scale unit sequence image obtained by segmentation processing and a scale unit position.

[0059] After obtaining the target scale value, the embodiment of the application can display it in the quality inspection system, and display it together with the time watermark on the image to be detected. In addition, the image to be detected and the target scale value are stored in the audit system as archive data of the current construction point, and as an audit basis for the engineering quality and quantity of the current construction point. Multiple construction points are linked to prevent construction fraud.

[0060] In an optional embodiment, the target scale value of the pole indicating the ruler in the image to be detected is obtained according to the ruler and pole segmentation result and the second scale unit sequence segmentation result, and includes:

[0061] According to the ruler and pole segmentation result, an intersection position of the ruler and the pole in the image to be detected is obtained;

[0062] According to the intersection position, a candidate scale unit in which the target scale value falls is determined in the second scale unit sequence segmentation result;

[0063] The target scale value is obtained according to the position of the candidate scale unit.

[0064] According to the ruler and pole segmentation result, the embodiment of the application obtains an intersection position G(x g ,y g ) of the ruler and the pole in the image to be detected, so as to determine which scale unit the target scale value falls in in the second scale unit sequence segmentation result, and the scale unit is the candidate scale unit. The target scale value is obtained according to the position of the candidate scale unit, and the detection accuracy is improved.

[0065] In an optional embodiment, the target scale value is composed of a target main scale value and a target secondary scale value;

[0066] The target scale value is obtained according to the position of the candidate scale unit, and includes:

[0067] The target main scale value is obtained according to the segmentation result;

[0068] The length of the candidate scale unit and the length difference between the position of the candidate scale unit and the intersection position are obtained according to the position of the candidate scale unit and the intersection position;

[0069] The proportion of the length difference in the length is calculated, and the target secondary scale value is obtained according to the proportion.

[0070] It can be understood that the target scale value includes a target main scale value and a target sub-scale value. For the target main scale value, it can be obtained by segmentation result; for the target sub-scale value, it can be obtained according to information of the candidate scale unit;

[0071] Specifically, assuming that the 4 corner point positions of the candidate scale unit are: In turn, the first corner point position, the second corner point position, the third corner point position, and the fourth corner point position of the candidate scale unit; wherein the first corner point position and the second corner point position are two corner point positions of the candidate scale unit along the scale direction; the third corner point position and the fourth corner point position are two corner point positions of the candidate scale unit along the scale direction; the first corner point position and the fourth corner point position are two corner point positions of the starting point of the candidate scale unit; the second corner point position and the third corner point position are two corner point positions of the end point of the candidate scale unit. For example, Figure 2 In which A, B, C, and D are in turn the first corner point position, the second corner point position, the third corner point position, and the fourth corner point position.

[0072] According to the following formula, the length of the candidate scale unit is calculated

[0073]

[0074] In the formula, represents the first corner point position of the candidate scale unit, represents the second corner point position of the candidate scale unit;

[0075] The length difference includes at least one of the following: a first length difference L G1 between the starting point position of the candidate scale unit and the intersection position; G2 a second length difference L G1 between the end point position of the candidate scale unit and the intersection position; and G2 L G1 Similar methods can be used for calculation, or after one of the length differences is obtained, the other length difference is obtained by subtracting the length difference.

[0076] According to the following formula, the first length difference L G1 is calculated:

[0077]

[0078] In the formula, represents the first corner point position of the candidate scale unit, represents the fourth corner point position of the candidate scale unit, (x g , y g ) represents the intersection position of the scale and the staff.

[0079] the second length difference L G2 the calculation of the first length difference L G1 Similar, here no longer.

[0080] the first length difference L G1 , the target sub-scale value is obtained as the second length difference L G2 , the target sub-scale value is obtained as

[0081] Finally, the target scale value is obtained as target main scale value + target sub-scale value, such as Figure 2 , the scale unit is 10, and the target scale value indicated is 30 + (7 / 10) * 10 = 37.

[0082] In an optional embodiment, the target main scale value is obtained according to the segmentation result, and the method comprises the following steps.

[0083] If the scale start point of the ruler is identified in the segmentation result, the number of scale units before the intersection position is obtained according to the second scale unit sequence segmentation result, and the target main scale value is obtained according to the number.

[0084] In the embodiment of the application, if the scale start point of the ruler in the to-be-detected image can be identified in the segmentation result (referring to the starting point of the ruler in the image being clear and visible, and the scale start point being able to be identified from the image), the number of scale units before the intersection is calculated according to the second scale unit sequence segmentation result, so that it is determined that the intersection falls within the z(th) scale unit (1≤z≤N), and the target scale value falls between the scale values in the range of (z-1) * 10 to z* 10, and the target main scale value u = (z-1) * scale unit unit, z represents the number of scale units before the intersection position.

[0085] In addition, the embodiment of the application also provides another scheme for determining the target main scale value. If the scale start point of the ruler in the to-be-detected image cannot be identified in the segmentation result, it is indicated that the range of the whole ten of the scale cannot be determined by calculating the number of scale units before the intersection. Then, the main scale value of the ruler indicated by the staff is directly detected from the image by using the image processing model. The detected main scale value is generally two, and the smaller main scale value is selected as the target main scale value of the ruler indicated by the staff in the to-be-detected image. For example, the main scale values before and after the intersection are 90 and 100, and 90 is taken as the target main scale value. The image processing model also comprises a third task for detecting the main scale value of the ruler indicated by the staff from the image. That is, by using the image processing model, the to-be-detected image is processed, and the main scale value detection result is also obtained. The main scale value detection result and the segmentation result constitute the processing result of the image processing model.

[0086] In an alternative embodiment, the step of verifying and correcting the first scale unit sequence segmentation result according to the distance between adjacent scale units to obtain a second scale unit sequence segmentation result comprises:

[0087] verifying whether there is a missed scale unit in the first scale unit sequence segmentation result according to the distance between adjacent scale units;

[0088] if not, taking the first scale unit sequence segmentation result as the second scale unit sequence segmentation result;

[0089] if yes, determining the adjacent abnormal scale units before and after the missed scale unit, obtaining the position of the missed scale unit according to the positions of the adjacent abnormal scale units, and correcting the first scale unit sequence segmentation result according to the position of the missed scale unit to obtain the second scale unit sequence segmentation result.

[0090] The embodiments of the present application verify whether there is a missed scale unit in the first scale unit sequence segmentation result according to the distance between adjacent scale units. If there is an abnormal distance, it means that there is a missed scale unit. The position of the missed scale unit is obtained according to the positions of the adjacent abnormal scale units before and after the missed scale unit, so as to correct the first scale unit sequence segmentation result.

[0091] In an alternative embodiment, the step of verifying whether there is a missed scale unit in the first scale unit sequence segmentation result according to the distance between adjacent scale units comprises:

[0092] calculating the first center point position of each scale unit in the first scale unit sequence segmentation result;

[0093] reordering all scale units in the first scale unit sequence segmentation result according to each first center point position;

[0094] performing least square fitting on the reordered first center point positions to obtain a center point straight line;

[0095] calculating the distance between adjacent first center point positions in the direction of the center point straight line, and judging whether there is a distance satisfying a preset condition.

[0096] The embodiments of the present application use the distance between the center point positions of adjacent scale units to represent the distance between adjacent scale units. According to the first scale unit sequence segmentation result, the 4 corner point positions of each scale unit {Q j j = 1 … N} are known. Thus, the first center point position of each scale unit can be calculated The N scale units are arranged in ascending order of the horizontal and vertical coordinates of the center points. The N first center point positions after sorting are fitted by the least square method to obtain a center point straight line y=ax+b, and the distance between adjacent center points is calculated along the direction of the center point straight line N-1 distances {d j , j=1…N-1} are obtained. According to the N-1 distances, it is determined whether there is a distance satisfying a preset condition, the verification of the missed scale unit is realized, and further, the adjacent abnormal scale units before and after can be determined according to the distance satisfying the preset condition.

[0097] For example, if one of the N-1 distances d2 is greater than a preset distance threshold, it means that d2 is an abnormal distance, indicating that a missed scale unit occurs between the second scale unit and the third scale unit, and the second scale unit and the third scale unit are adjacent abnormal scale units, so that the missed scale unit is supplemented in the adjacent abnormal scale units, and the correction of the first scale unit sequence segmentation result is realized; if there is no abnormal distance in the N-1 distances, the first scale unit sequence segmentation result does not have a missed scale unit, and the first scale unit sequence segmentation result is directly used as the second scale unit sequence segmentation result for subsequent steps.

[0098] In an optional embodiment, the position of the missed scale unit is obtained according to the positions of the adjacent abnormal scale units before and after, including:

[0099] The second center point position of the missed scale unit is obtained according to the positions of the adjacent abnormal scale units before and after;

[0100] The position of the missed scale unit is obtained according to the second center point position.

[0101] It can be understood that a scale unit is a whole containing multiple scale values, and the target is large, in order to ensure the detection accuracy, the embodiment of the present application considers the case that there is a missed scale unit between adjacent abnormal scale units, and the center point position between the adjacent abnormal scale units before and after is taken as the second center point position of the missed scale unit, so as to obtain the position of the missed scale unit.

[0102] Specifically, the second center point position of the missed scale unit is calculated according to the following formula:

[0103]

[0104] In the formula, is the second center point position of the missed scale unit, is the center point position of the first abnormal scale unit, a center point position of a second abnormal scale unit adjacent to the first abnormal scale unit;

[0105] The position of the missed scale unit is calculated according to the following formula:

[0106]

[0107] wherein, are respectively four corner point positions of the missed scale unit, are respectively four corner point positions of the first abnormal scale unit. Of course, the position of the missed scale unit can also be calculated using the four corner point positions of the second abnormal scale unit, and the process is similar to that of the first abnormal scale unit, which will not be described here.

[0108] In an optional embodiment, the loss function of the image processing model is composed of at least a first loss function of the first task and the second task, and the first loss function is used to measure the difference between the model segmentation result and the real label.

[0109] The embodiments of the present application combine the first loss function L seg The first loss function L seg is used to train the first task and the second task. The cross-entropy loss is used to realize the segmentation of the scale, the staff and the scale unit, and the target is to minimize the error between the model segmentation result and the real label ground truth.

[0110]

[0111] wherein, y true represents the real label, y pred represents the model segmentation result.

[0112] In an optional embodiment, the multi-task further includes a third task for detecting the main scale value of the staff indicating the scale from the image, and the loss function of the image processing model is further composed of a second loss function of the third task, and the second loss function is composed of a first sub-loss function, a second sub-loss function and a third sub-loss function; the first sub-loss function is used to measure the difference between the main scale value prediction category and the main scale value real category, the second sub-loss function is used to determine whether the target detection frame includes the main scale value, and the fourth loss function is used to measure the difference between the main scale value detection frame and the main scale value annotation frame.

[0113] The embodiments of the present application optimize the loss function of the image processing model by combining the first sub-loss function L class , the second sub-loss function L obj and the third sub-loss function L bboxThe third task is trained to improve the target recognition effect.

[0114] Further, the first sub-loss function L class is a classification loss, focal loss can be selected to enable the image processing model to give priority to difficult samples, the second sub-loss function L obj is a confidence loss, and focal loss is also selected to enable the image processing model to give priority to difficult samples, and the third sub-loss function L bbox is a bounding box regression loss, which considers the distance, overlap rate, size, and prediction probability of the main scale value label box when calculating the loss.

[0115] The loss function L all of the entire image processing model can be defined as a weighted sum function:

[0116] L all = γ2L seg + γ1L det = γ2L seg + γ1*(α1L class + α2L obj + α3L bbox )

[0117] Wherein, γ1 represents a preset coefficient of the first loss function, L seg represents the first loss function, γ2 represents a preset coefficient of the second loss function, L det represents the second loss function; α1 represents a preset coefficient of the first sub-loss function, L class represents the first sub-loss function, α2 represents a preset coefficient of the second sub-loss function, L obj represents the second sub-loss function, and α3 represents a preset coefficient of the third sub-loss function, L bbox represents the third sub-loss function. Of course, other ways can be used, not limited to this formula.

[0118] In an optional embodiment, the image processing model is composed of a shared encoder and at least two decoders; the at least two decoders include a first decoder and a second decoder, the first decoder is used to perform the first task, and the second decoder is used to perform the second task.

[0119] For example, a certain number of images of the width and height of pits, trenches, and pedestals are taken by a portable terminal device, and the ruler, pole, and scale unit are labeled to construct a training set, i.e., an image sample set and a test set, and the training set is used for multi-task training to obtain a trained image processing model. In the training process, the general deep neural network training principle is followed, and the learning rate, batch size, and other parameters are adjusted to optimize the performance.

[0120] In order to solve the challenge of small sample problem in practical engineering to network training, an image processing model composed of a shared encoder and at least two decoders is built. The shared encoder is used to extract features from the input image. Further, the image sample set includes a global image (Whole Image) and a local image (Partial Image). The global image is used as a segmentation flow (Segmentation Flow), and the local image is used as a detection flow (Detec Flow). The convolutional neural network shared by the segmentation flow and the detection flow can better learn the global features and detail features of the image, and can effectively improve the precision of the small sample task. One of the decoders (the first decoder) is used to perform the first task, and the other decoder (the second decoder) is used to perform the second task.

[0121] Further, the at least two decoders further include a third decoder, which is used to perform the third task. Here, the image processing model is composed of a shared encoder and three decoders.

[0122] As Figure 3 , the shared encoder is composed of a "backbone network" and a "Neck network". The backbone network can extract features from the input image, and adopts a design similar to YOLOPv2 and E-ELAN. Group convolution is used as the backbone network structure, which can effectively solve the gradient redundancy problem in the optimization process, support efficient propagation and reuse of features, and meet the real-time requirements of the model. The Neck network is used to fuse the features generated by the preceding backbone network, which is composed of a spatial pyramid pooling structure (SPP) and a feature pyramid network (FPN) structure. The SPP generates and fuses features of different scales, and the FPN fuses features of different semantic levels, ensuring that the generated features contain multi-scale and multi-level semantic information.

[0123] For the decoder, two categories are divided. The first decoder and the second decoder are decoders for segmentation tasks (specifically, the first task, the second task), and the segmentation head is a specific implementation of this type of decoder; the third decoder is a decoder for the target detection task (specifically, the third task), and the detection head is a specific implementation of this type of decoder. For the segmentation head, the output features of the neck network are directly input into the segmentation head, and the prediction results are obtained through multiple upsampling. For the detection head (Detect Head), a multi-scale detection strategy based on anchor points is adopted, and features with multi-scale information are used for detection. Each anchor point on a multi-scale feature map represents a prior box with different proportions, and then the detection head predicts the position offset, the box size, and the class label.

[0124] The scale value detection method provided by the embodiment of the present application effectively avoids the problem of low detection precision caused by too small scale values by processing multiple scale values as a scale unit for image processing, and improves the recognition detection precision; by verifying and correcting the first scale unit sequence segmentation result, the scale missing detection phenomenon caused by factors such as occlusion, shooting angle, and image blur can be avoided, and the accuracy and precision of scale value detection are improved. The embodiment of the present application has strong robustness, and has high precision for scale value detection under various complex conditions such as different backgrounds, different types of scales and poles, different shooting distances, and different angles, and is suitable for various complex engineering construction scenes.

[0125] Referring to Figure 4 , Figure 4 is a structural block diagram of a scale value detection device 10 provided by the embodiment of the present application, and the scale value detection device 10 comprises:

[0126] The acquisition module 11 is configured to acquire a to-be-detected image.

[0127] The processing module 12 is configured to process the to-be-detected image by using a pre-trained image processing model to obtain a segmentation result; the segmentation result comprises a scale and pole segmentation result and a first scale unit sequence segmentation result; the image processing model is obtained by using an image sample set for multi-task training, the multi-task at least comprises a first task and a second task, the first task is used for extracting a scale and a pole from an image, and the second task is used for extracting a first scale unit sequence from an image; the first scale unit sequence comprises at least one scale unit, and each scale unit comprises a plurality of scale values.

[0128] The correction module 13 is configured to verify and correct the first scale unit sequence segmentation result according to the distance between adjacent scale units to obtain a second scale unit sequence segmentation result.

[0129] The detection module 14 is configured to obtain a target scale value of a staff indicating a scale in the image to be detected according to the scale-staff segmentation result and the second scale unit sequence segmentation result.

[0130] Optionally, the detection module 14 is specifically configured to:

[0131] obtain a position of an intersection of the scale and the staff in the image to be detected according to the scale-staff segmentation result;

[0132] determine a candidate scale unit in which the target scale value falls in the second scale unit sequence segmentation result according to the position of the intersection;

[0133] obtain the target scale value according to a position of the candidate scale unit.

[0134] Optionally, the target scale value is composed of a target primary scale value and a target secondary scale value.

[0135] The detection module 14 is specifically configured to:

[0136] obtain the target primary scale value according to the segmentation result.

[0137] obtain a length of the candidate scale unit and a length difference between the position of the candidate scale unit and the position of the intersection according to the position of the candidate scale unit and the position of the intersection.

[0138] calculate a proportion of the length difference in the length, and obtain the target secondary scale value according to the proportion.

[0139] Optionally, the detection module 14 is specifically configured to:

[0140] if a scale starting point of the scale is recognized in the segmentation result, obtain a number of scale units before the position of the intersection according to the second scale unit sequence segmentation result, and obtain the target primary scale value according to the number.

[0141] Optionally, the correction module 13 is specifically configured to:

[0142] verify whether there is a missed scale unit in the first scale unit sequence segmentation result according to a distance between adjacent scale units.

[0143] if not, take the first scale unit sequence segmentation result as the second scale unit sequence segmentation result.

[0144] If the missed calibration unit exists, adjacent abnormal calibration units before and after the missed calibration unit are determined, the position of the missed calibration unit is obtained according to the positions of the adjacent abnormal calibration units, and the first calibration unit sequence segmentation result is corrected according to the position of the missed calibration unit to obtain the second calibration unit sequence segmentation result.

[0145] Optionally, the correction module 13 is specifically configured to:

[0146] The first center point position of each calibration unit in the first calibration unit sequence segmentation result is calculated.

[0147] Each of the first center point positions is used to reorder all the calibration units in the first calibration unit sequence segmentation result.

[0148] The reordered first center point positions are fitted by a least square method to obtain a center point straight line.

[0149] The distance between the adjacent reordered first center point positions is calculated in the direction of the center point straight line, and it is determined whether there is a distance meeting a preset condition.

[0150] Optionally, the correction module 13 is specifically configured to:

[0151] The second center point position of the missed calibration unit is obtained according to the positions of the adjacent abnormal calibration units before and after the missed calibration unit.

[0152] The position of the missed calibration unit is obtained according to the second center point position.

[0153] Optionally, the loss function of the image processing model is composed of at least a first loss function of the first task and the second task, and the first loss function is used to measure the difference between the model segmentation result and the real label.

[0154] Optionally, the multi-task further includes a third task, the third task is used to detect the main calibration value of the staff indicating the scale from the image, the loss function of the image processing model is further composed of a second loss function of the third task, and the second loss function is composed of a first sub-loss function, a second sub-loss function and a third sub-loss function; the first sub-loss function is used to measure the difference between the main calibration value prediction category and the main calibration value real category, the second sub-loss function is used to determine whether the target detection frame includes the main calibration value, and the fourth loss function is used to measure the difference between the main calibration value detection frame and the main calibration value annotation frame.

[0155] Optionally, the image processing model is composed of a shared encoder and at least two decoders; the at least two decoders include a first decoder and a second decoder, the first decoder is used for performing the first task, and the second decoder is used for performing the second task.

[0156] It is worth noting that the working processes of various modules in the scale value detection apparatus 10 described in the embodiments of the present application can refer to the working processes of the scale value detection method described in the above embodiments, which will not be described here.

[0157] The scale value detection apparatus 10 provided in the embodiments of the present application effectively avoids the problem of low detection precision caused by too small scale values by performing image processing on multiple scale values as a scale unit, thereby improving the recognition detection precision; by verifying and correcting the first scale unit sequence segmentation result, the scale detection accuracy and precision can be improved by avoiding the scale detection missing phenomenon caused by factors such as occlusion, shooting angle, image blur, etc. The embodiments of the present application have strong robustness and high precision for scale value detection under various complex conditions such as different backgrounds, different types of scales and poles, different shooting distances and angles, etc. and are suitable for various complex engineering construction scenes.

[0158] In addition, the embodiments of the present application also provide a computer readable storage medium, which includes a stored computer program; wherein the computer program controls the device where the computer readable storage medium is located to perform the scale value detection method as described in any of the above embodiments when running.

[0159] In addition, the embodiments of the present application also provide a computer program product, which includes computer programs / instructions, and the computer programs / instructions are executed by a processor to implement the scale value detection method as described in any of the above embodiments.

[0160] Referring to Figure 5 , Figure 5 is a structural block diagram of a scale value detection device 20 provided in the embodiments of the present application, the scale value detection device 20 includes a processor 21, a memory 22, and a computer program stored in the memory 22 and executable on the processor 21. The processor 21 implements the steps in the above scale value detection method embodiments when executing the computer program. Alternatively, the processor 21 implements the functions of each module / unit in the above various apparatus embodiments when executing the computer program.

[0161] For example, the computer program can be divided into one or more modules / units, which are stored in the memory 22 and executed by the processor 21 to complete the present application. The one or more modules / units can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program in the scale value detection device 20.

[0162] The scale value detection device 20 can include, but is not limited to, a processor 21, a memory 22. Those skilled in the art can understand that the schematic diagram is only an example of the scale value detection device 20, and does not constitute a limitation on the scale value detection device 20, and can include more or fewer components than the diagram, or combine certain components, or different components, for example, the scale value detection device 20 can also include an input / output device, a network access device, a bus, etc.

[0163] The processor 21 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The processor 21 is the control center of the scale value detection device 20, which connects all parts of the scale value detection device 20 through various interfaces and lines.

[0164] The memory 22 can be used to store the computer programs and / or modules, and the processor 21 realizes various functions of the scale value detection device 20 by running or executing the computer programs and / or modules stored in the memory 22, and calling the data stored in the memory 22. The memory 22 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.), and the like; and the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), and the like. In addition, the memory 22 can include a high-speed random access memory, and can also include a nonvolatile memory, for example, a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state memory devices.

[0165] When the modules / units of the scale value detection device 20 are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware, and the computer program can be stored in a computer readable storage medium. When the processor 21 executes the computer program, the steps of the above-mentioned various method embodiments can be realized. The computer program includes computer program code, which can be in the form of source code, object code, an executable file, or some intermediate form. The computer readable medium can include any entity or device capable of carrying the computer program code, a recording medium, a U disk, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0166] It should be noted that the apparatus embodiments described above are merely illustrative, and the units described as separate units can or can not be physically separate, and the units shown as units can or can not be physical units, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment according to actual needs. In addition, the connection relationship between the modules in the apparatus embodiment provided in the present application indicates that there is a communication connection between them, which can be implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement it without creative labor.

[0167] The above is the preferred embodiment of the present application. It should be noted that those skilled in the art can make several improvements and refinements without departing from the principles of the present application, and these improvements and refinements are also considered within the scope of protection of the present application.

Claims

1. A method for detecting scale values, characterized in that, include: Acquire the image to be detected; The image to be detected is processed by a pre-trained image processing model to obtain a segmentation result. The segmentation result includes a ruler and pole segmentation result and a first scale unit sequence segmentation result. The image processing model is trained using an image sample set for multi-task training. The multi-task includes at least a first task and a second task. The first task is used to extract the ruler and pole from the image, and the second task is used to extract the first scale unit sequence from the image. The first scale unit sequence includes at least one scale unit, and each scale unit consists of multiple scale values. Based on the distance between adjacent scale units, the segmentation result of the first scale unit sequence is verified and corrected to obtain the segmentation result of the second scale unit sequence. Based on the segmentation results of the ruler and the pole and the segmentation results of the second scale unit sequence, the target scale value of the ruler in the image to be detected is obtained.

2. The scale value detection method as described in claim 1, characterized in that, The step of obtaining the target scale value of the scale indicating the ruler in the image to be detected based on the scale and pole segmentation results and the second scale unit sequence segmentation results includes: Based on the segmentation results of the ruler and the pole, the position of the intersection point of the ruler and the pole in the image to be detected is obtained; Based on the intersection point position, the candidate scale unit in the second scale unit sequence segmentation result is determined to be where the target scale value falls; The target scale value is obtained based on the position of the candidate scale unit.

3. The scale value detection method as described in claim 2, characterized in that, The target scale value consists of a target main scale value and a target subscale value; The step of obtaining the target scale value based on the position of the candidate scale unit includes: Based on the segmentation results, the target principal scale value is obtained; Based on the position of the candidate scale unit and the position of the intersection point, the length of the candidate scale unit and the length difference between the position of the candidate scale unit and the position of the intersection point are obtained; Calculate the proportion of the length difference in the length, and obtain the target subscale value based on the proportion.

4. The scale value detection method as described in claim 3, characterized in that, The step of obtaining the target principal scale value based on the segmentation result includes: If the starting point of the scale is identified in the segmentation result, the number of scale units before the intersection position is obtained according to the segmentation result of the second scale unit sequence, and the target main scale value is obtained according to the number.

5. The scale value detection method as described in claim 1, characterized in that, The step of verifying and correcting the first scale unit sequence segmentation result based on the distance between adjacent scale units to obtain the second scale unit sequence segmentation result includes: Based on the distance between adjacent scale units, the first scale unit sequence segmentation result is checked to see if there are any missed scale units. If it does not exist, the first scale unit sequence segmentation result is used as the second scale unit sequence segmentation result; If present, identify the abnormal scale units adjacent to the missed detection scale unit; based on the positions of the adjacent abnormal scale units, obtain the position of the missed detection scale unit; correct the first scale unit sequence segmentation result based on the position of the missed detection scale unit to obtain the second scale unit sequence segmentation result.

6. The scale value detection method as described in claim 5, characterized in that, The step of verifying whether there are any missed scale units in the segmentation result of the first scale unit sequence based on the distance between adjacent scale units includes: Calculate the position of the first center point of each scale unit in the first scale unit sequence segmentation result; Based on the position of each of the first center points, all scale units in the first scale unit sequence segmentation result are reordered. The position of the first center point after sorting is fitted by least squares to obtain the center point line; Calculate the distance between adjacent first center point positions after sorting along the direction of the straight line from the center point, and determine whether there is a distance that meets the preset conditions.

7. The scale value detection method according to any one of claims 5, characterized in that, The step of determining the position of the missed detection scale unit based on the positions of the adjacent abnormal scale units includes: The position of the second center point of the missed detection scale unit is obtained based on the positions of the adjacent abnormal scale units. The position of the missed detection scale unit is obtained based on the position of the second center point.

8. The method for detecting scale values ​​as described in any one of claims 1-7, characterized in that, The loss function of the image processing model consists of at least the first loss function of the first task and the first loss function of the second task, wherein the first loss function is used to measure the difference between the model segmentation result and the real label.

9. The scale value detection method as described in claim 8, characterized in that, The multi-task further includes a third task, which is used to detect the main scale value of the ruler from the image. The loss function of the image processing model is also composed of the second loss function of the third task. The second loss function is composed of a first sub-loss function, a second sub-loss function, and a third sub-loss function. The first sub-loss function is used to measure the difference between the predicted category of the main scale value and the true category of the main scale value. The second sub-loss function is used to determine whether the target detection box includes the main scale value. The fourth loss function is used to measure the difference between the main scale value detection box and the main scale value annotation box.

10. The method for detecting scale values ​​as described in any one of claims 1-7, characterized in that, The image processing model consists of a shared encoder and at least two decoders; the at least two decoders include a first decoder and a second decoder, the first decoder being used to perform the first task and the second decoder being used to perform the second task.

11. A scale value detection device, characterized in that, include: The acquisition module is used to acquire the image to be detected; The processing module is used to process the image to be detected using a pre-trained image processing model to obtain a segmentation result. The segmentation result includes a ruler and pole segmentation result and a first scale unit sequence segmentation result. The image processing model is obtained by multi-task training using an image sample set. The multi-task includes at least a first task and a second task. The first task is used to extract the ruler and pole from the image, and the second task is used to extract the first scale unit sequence from the image. The first scale unit sequence includes at least one scale unit, and each scale unit consists of multiple scale values. The correction module is used to verify and correct the segmentation result of the first scale unit sequence based on the distance between adjacent scale units, so as to obtain the segmentation result of the second scale unit sequence. The detection module is used to obtain the target scale value of the scale indicating the scale in the image to be detected based on the scale and pole segmentation results and the second scale unit sequence segmentation results.

12. A scale value detection device, characterized in that, The device includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the scale value detection method as described in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program; wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the scale value detection method as described in any one of claims 1 to 10.

14. A computer program product, characterized in that, Includes a computer program / instruction that, when executed by a processor, implements the scale value detection method as described in any one of claims 1 to 10.

Images