First character detection and recognition method and system based on 3D vision technology

Abstract

The application provides a first-tire character detection and recognition method and system based on 3D vision technology, wherein the method comprises the following steps: acquiring point cloud data of a tire side character area; pre-processing the point cloud data, including filtering out abnormal value point cloud data and unfolding the curved surface point cloud of the tire surface into a plane point cloud after fitting; converting the pre-processed point cloud data into a depth image; labeling the depth image and generating a file corresponding to the labeling information; training a character detection network according to coordinate data samples of the character area to obtain a character detection model; training a character recognition network according to character data samples of the character area to obtain a character recognition model; detecting and recognizing the tire side character area through the character detection model and the character recognition model, and displaying the detection result of the character detection model and the recognition result of the character recognition model in the original image respectively.

Description

Technical Field

[0001] This invention relates to the field of 3D vision and image processing technology, specifically to a method and system for detecting and recognizing first-time birth characters based on 3D vision technology. Background Technology

[0002] Tires are an essential component of automobiles, and the lettering on the tire sidewall indicates important information such as tire specifications, load index, and speed rating. During the initial die-cutting process, omissions, overprints, or misprints of lettering will result in defective or scrap products, causing significant economic losses for tire manufacturers. Therefore, the inspection of lettering on the first tire during the initial die-cutting process is particularly important.

[0003] Currently, most tire character detection methods rely on manual labor or 2D vision. Manual detection is time-consuming and labor-intensive, increasing worker fatigue and production costs, while also resulting in poor accuracy and low efficiency. 2D vision-based tire character detection suffers from limitations. Because the tire tread is black, the image has a low grayscale value under the same optical conditions, lacking significant contrast. Furthermore, since 2D vision technology depends on measuring the contrast of the object, it cannot measure black markings on a black background. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a method and system for first-time birth character detection and recognition based on 3D vision technology, which can reduce production costs for enterprises and improve the accuracy and efficiency of first-time birth character detection.

[0005] The technical solution adopted in this invention is as follows:

[0006] A method for character detection and recognition on a tire sidewall based on 3D vision technology includes the following steps: acquiring point cloud data of the character region on the tire sidewall; preprocessing the point cloud data, including filtering outlier point cloud data and fitting and unfolding the curved point cloud of the tire sidewall into a planar point cloud; converting the preprocessed point cloud data into a depth image; annotating the depth image and generating a file corresponding to the annotation information, wherein the file includes coordinate data samples of the character region and character data samples within the character region; training a character detection network based on the coordinate data samples of the character region to obtain a character detection model; training a character recognition network based on the character data samples of the character region to obtain a character recognition model; detecting and recognizing the character region on the tire sidewall using the character detection model and the character recognition model, and displaying the detection results of the character detection model and the recognition results of the character recognition model in the original image respectively.

[0007] The preprocessed point cloud data is converted into a depth image, specifically including: converting the preprocessed point cloud data into a grayscale image; converting the height information on the Z-axis into depth information by using the height value difference between the tire character and non-character regions in the Z-axis direction; and converting the point cloud data of the tire character pairs into a depth image.

[0008] The character detection network is a ResNet50 network based on the DB algorithm.

[0009] The DB algorithm is used to train the ResNet50 network, specifically including: the depth image is passed through the ResNet50 network with a feature pyramid structure, and a feature layer is generated by upsampling; a dynamic threshold probability map and a text probability map are predicted by the feature layer; a DB binary map is formed by the dynamic threshold probability map and the text probability map, and a text box is generated by combining the DB binary map with the label.

[0010] The character recognition network is a CRNN (Convolutional Recurrent Neural Network) network based on the CTC (connectionist temporal classification) algorithm. The CRNN network includes convolutional layers, recurrent layers, and transcriptional layers, wherein the recurrent layers and the transcriptional layers are connected.

[0011] A tire character detection and recognition system based on 3D vision technology includes: an acquisition module for acquiring point cloud data of the character region on the tire sidewall; a preprocessing module for preprocessing the point cloud data, including filtering out outlier point cloud data and fitting and unfolding the curved point cloud of the tire sidewall into a planar point cloud; an image processing module for converting the preprocessed point cloud data into a depth image; and an annotation module for annotating the depth image and generating a file corresponding to the annotation information, wherein the file includes coordinate data samples of the character region. The image contains character data samples within the character region; a first training module, used to train a character detection network based on the coordinate data samples of the character region to obtain a character detection model; a second training module, used to train a character recognition network based on the character data samples of the character region to obtain a character recognition model; and a display module, used to detect and recognize the character region on the tire side using the character detection model and the character recognition model, and to display the detection results of the character detection model and the recognition results of the character recognition model in the original image.

[0012] The preprocessed point cloud data is converted into a depth image, specifically including: converting the preprocessed point cloud data into a grayscale image; converting the height information on the Z-axis into depth information by using the height value difference between the tire character and non-character regions in the Z-axis direction; and converting the point cloud data of the tire character pairs into a depth image.

[0013] The character detection network is a ResNet50 network based on the DB algorithm.

[0014] The DB algorithm is used to train the ResNet50 network, specifically including: the depth image is passed through the ResNet50 network with a feature pyramid structure, and a feature layer is generated by upsampling; a dynamic threshold probability map and a text probability map are predicted by the feature layer; a DB binary map is formed by the dynamic threshold probability map and the text probability map, and a text box is generated by combining the DB binary map with the label.

[0015] The character recognition network is a CRNN network based on the CTC algorithm, wherein the CRNN network includes a convolutional layer, a recurrent layer and a transcriptional layer, wherein the recurrent layer and the transcriptional layer are connected.

[0016] The beneficial effects of this invention are:

[0017] This invention acquires point cloud data of the character region on the tire sidewall, preprocesses the point cloud data, converts the preprocessed point cloud data into a depth image, and annotates the depth image to generate coordinate data samples and character data samples of the character region. Based on the coordinate data samples and character data samples, a detection network and a recognition network are trained to obtain a detection model and a recognition model. Finally, the detection and recognition results are displayed in the original image. This can reduce the production cost of enterprises and improve the accuracy and efficiency of character detection in first tires. Attached Figure Description

[0018] Figure 1 This is a flowchart of a first-time character detection and recognition method based on 3D vision technology according to an embodiment of the present invention;

[0019] Figure 2 This is a fitted cylindrical surface diagram of the tire surface point cloud according to an embodiment of the present invention;

[0020] Figure 3 This is a depth image of the point cloud data of a tire character according to an embodiment of the present invention;

[0021] Figure 4 This is an interface diagram of the tire character detection and recognition results according to an embodiment of the present invention;

[0022] Figure 5This is a block diagram of the first-time character detection and recognition system based on 3D vision technology according to an embodiment of the present invention;

[0023] Figure 6 This is a schematic diagram of the left side of a first-time fetus character detection device according to an embodiment of the present invention;

[0024] Figure 7 This is a schematic diagram of the right side of a first-time character detection device according to an embodiment of the present invention. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] Figure 1 This is a flowchart of a first-time character detection and recognition method based on 3D vision technology according to an embodiment of the present invention.

[0027] like Figure 1 As shown, the first-time character detection and recognition method based on 3D vision technology according to an embodiment of the present invention includes the following steps:

[0028] S1, acquire point cloud data of the character area on the tire sidewall.

[0029] In one embodiment of the present invention, a 3D line-scanning laser camera can be placed on a camera mount facing the side of the tire, and the first tire character detection device can be activated to acquire point cloud data of the character area on the side of the tire, wherein the tire can be fixed on the tire mount.

[0030] In one embodiment of the present invention, the first tire character detection device may include a line structured light three-dimensional measurement device, a tire fixing and rotating device, and a base. The line structured light three-dimensional measurement device consists of a column and a line structured light three-dimensional measurement instrument. The column stands on the tire base and is fixed to the tire base at the bottom with hexagonal bolts. The line structured light measurement instrument is fixed to the column with screws.

[0031] In one embodiment of the present invention, the tire fixing and rotating device comprises a tire fixing bracket, a stepper motor, rollers, and a tire. The tire fixing bracket is fixed to the tire base with hexagonal bolts. One end of the rotating shaft in the middle of the tire is fixed to the wheel center cover of the tire, and the other end passes through the shaft hole of the bracket and is fixed to the bracket by a bearing and a flange at the top of the bracket shaft hole. A stepper motor is mounted on the tire fixing bracket. When the start button is pressed, the stepper motor rotates at a speed set by the system, and the roller fixed to one end of the motor also rotates accordingly. The friction between the inner wall of the tire and the roller drives the tire to rotate.

[0032] S2 preprocesses the point cloud data, including filtering out outlier point cloud data and fitting the curved point cloud of the tire sidewall into a planar point cloud.

[0033] In one embodiment of the present invention, since the character region in the acquired tire point cloud data is mainly concentrated within a fixed range along the y-axis, a point cloud pass-through filtering algorithm can be used to process the point cloud data. Specifically, the values ​​of parameters Ymin and Ymax can be set to filter out point cloud data outside the range of (Ymin, Ymax). A point cloud statistical filtering algorithm can be used to calculate the average distance between each point in the point cloud data and its ten neighboring points. Points with an average distance greater than one standard deviation of the standard distance can be considered outliers and removed from the point cloud data.

[0034] In one embodiment of the present invention, such as Figure 2 As shown, a spatial cylindrical surface of the tire surface point cloud can be fitted first, and then each point can be unfolded into a planar point cloud along the X-axis. Specifically, firstly, due to the geometric properties of the cylindrical surface, it is known that the cylindrical surface is a collection of point cloud data equidistant from the center. Therefore, the seven parameters of the cylindrical surface can be determined, which are the coordinates of a point on the axis ( , , ), the axis direction vector (a, b, c), and the radius R of the cylinder. Wherein, the coordinates of a point on the axis (a, b, c) , , The coordinates of all points are the average of the coordinates of all points, and the direction vector is a unit vector pointing in the positive direction. Furthermore, it is stipulated that a > 0; if a = 0, then b > 0; if a = 0 and b = 0, then c > 0, meaning that a, b, and c cannot all be 0 simultaneously. Secondly, based on the error equation, relatively close initial values ​​can be assigned to the fitting parameters, where the error equation is:

[0035]

[0036] in,( , , Let be the coordinates of any point on the axis, and i < R. Linearizing the error equation, we have: ,in,

[0037]

[0038]

[0039]

[0040] in, For any observation point ( , , The perpendicular distance from the axis to the actual radius is then used. The solution can then be obtained through least-squares iterative calculation using the indirect difference of squares method with additional constraints. , , The system uses seven parameters, a, b, c, and R, to obtain a fitted cylindrical surface of the tire sidewall point cloud. Additional constraints are as follows:

[0041]

[0042] .

[0043] Finally, we can set the coordinates of any point A in the point cloud data on the tire surface as ( , , ), take the coordinates of a point B on the fitted cylindrical surface that is coplanar with point A as ( , , Expand point A as point, The coordinates of the point are ( , , ).in, θ is the angle between line AB and the Z-axis. The arc length can be calculated using the arc length formula. .like < ,but ;like > ,but .in, R is the radius of the fitted cylindrical surface. Let be the distance between point A and point B. By repeating the above steps to calculate all points in the point cloud data, the point cloud surface can be unfolded into a planar point cloud.

[0044] S3 converts the preprocessed point cloud data into a depth image.

[0045] In one embodiment of the present invention, the preprocessed point cloud data can first be converted into a grayscale image. Specifically, a point M in the point cloud data can be selected. , , The value after grayscale conversion of point M is ( , (grey). Among them, Point , Then, by analyzing the height difference between the tire character and non-character regions along the Z-axis, the height information on the Z-axis is converted into depth information. Specifically, the Z-values ​​of all point cloud data are traversed to obtain the height information along the Z-axis in the point cloud data. and , Finally, all point cloud data are transformed according to the above steps to obtain a depth image of the tire character point cloud data, such as... Figure 3 As shown.

[0046] S4, annotate the depth image and generate a file corresponding to the annotation information, wherein the file includes coordinate data samples of the character region and character data samples within the character region.

[0047] In one embodiment of the present invention, a rectangular ROI region can be set to select the character region. At the same time, a file corresponding to the annotation information can be generated. The file can be a txt file, and the file can store the path address of the depth image, the coordinates of the four vertices of each rectangular ROI region, and the character information selected by each ROI region.

[0048] S5. The character detection network is trained based on the coordinate data samples of the character region to obtain the character detection model.

[0049] In one embodiment of the present invention, a character detection network can be trained using a ResNet50 network based on the DB algorithm. Specifically, the training may include: passing a depth image through a ResNet50 network with a feature pyramid structure, generating a feature layer by upsampling, predicting a dynamic threshold probability map and a text probability map through the feature layer, forming a DB binary map through the dynamic threshold probability map and the text probability map, and generating a text box based on the DB binary map and the label.

[0050] Specifically, to acquire receptive fields of different sizes and improve the detection accuracy for different feature font regions, binarization can be inserted into the segmentation network for joint optimization. This allows the network to adaptively predict the threshold for each pixel in the image, thus completely distinguishing foreground and background pixels. First, the input image can be processed through a ResNet50 backbone network with convolutional kernels of different sizes, outputting feature maps of 1 / 2, 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image size. An FPN structure is then added to construct a feature pyramid structure. For example, the 1 / 32 feature map can be upsampled to 1 / 16 of its original size and fused with the 1 / 16 feature map to obtain a new feature map of 1 / 16 of the original image size. This results in four feature maps of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image size. The four feature maps of sizes 1 / 4, 1 / 8, 1 / 16, and 1 / 32 can be upsampled to 1 / 4 size respectively, and then concatenated with the data from the four feature maps through a concat layer to generate a feature layer.

[0051] Furthermore, the aforementioned feature layer is first processed through 1 A convolutional layer of size 1 is used, compressing the channels to 1 / 4 of the input. After passing through Batch Normalization (BN) and ReLU, the resulting feature map has a shape of (batch, 64, 1 / 4W, 1 / 4H). This feature map is then deconvolved with a kernel of size (2, 2), resulting in a feature map with a shape of (batch, 256, 1 / 2W, 1 / 2H), which is half the size of the original image. Another deconvolution operation is performed, outputting a feature map with 1 channel, now with a shape of (batch, W, H), the same size as the original image. This is then passed through a sigmoid activation function, outputting a text probability map P and a dynamic threshold value for the threshold probability map. The threshold value is then used in the binarization formula to obtain the binarized threshold probability map T. The binarization formula is:

[0052]

[0053] in, This is an approximate binary image, where k is the magnification factor, typically taken as 50, and P and T are the calculated results.

[0054] Finally, P and T can be jointly calculated using the differentiable binarization formula to obtain the final DB binarized probability map, and a text box can be generated based on the DB binarized probability map and the label.

[0055] S6. Train the character recognition network based on character data samples from the character region to obtain a character recognition model.

[0056] In one embodiment of the present invention, the character recognition network may be a CRNN network based on the CTC algorithm. The CRNN network includes a convolutional layer, a recurrent layer and a transcriptional layer, wherein the recurrent layer and the transcriptional layer are connected.

[0057] Specifically, the convolutional layer extracts feature sequences from the input image. A ResNet34 network can be used to extract these features before feeding them into the CRNN text recognition algorithm's network structure. The recurrent layer predicts the label distribution of the feature sequences obtained from the convolutional layer. A bidirectional LSTM recurrent neural network can be used. LSTM (Long Short-Term Memory) is a special type of RNN (Recurrent Neural Network) designed to address the long-term dependency problem in RNNs. Ordinary RNNs suffer from gradient minimization or gradient explosion, failing to capture sufficient contextual information. Therefore, the LSTM used in CRNN allows for the capture of long-range dependencies. The transcription layer transforms the label distribution obtained from the recurrent layer into the final recognition result through deduplication and integration. The CTC algorithm can be used to parse the result. CTC is a loss calculation method that introduces blank characters to address the issue of missing characters in certain positions and rapidly calculates gradients through recursion. First, a CNN extracts image convolutional features. Then, an LSTM further extracts sequence features from the image convolutional features. Finally, CTC is introduced to address the character misalignment problem during training.

[0058] S7 uses a character detection model and a character recognition model to detect and recognize the character area on the side of the tire, and displays the detection results of the character detection model and the recognition results of the character recognition model in the original image respectively.

[0059] In one embodiment of the present invention, such as Figure 4 As shown, the detected character regions and the recognized character results can be displayed on the software interface. Based on the detection and recognition threshold, outputs with results below the threshold can be masked, and only results above the threshold can be output.

[0060] According to the embodiment of the present invention, the first tire character detection and recognition method based on 3D vision technology acquires point cloud data of the character area on the tire sidewall, preprocesses the point cloud data and converts it into a depth image, annotates the depth image and generates coordinate data samples and character data samples, trains the network based on the coordinate data samples and character data samples to obtain a character detection and recognition model, and detects and recognizes the character area on the tire sidewall. Thus, it can reduce the production cost of enterprises and improve the accuracy and efficiency of first tire character detection.

[0061] To realize the first-time birth character detection and recognition method based on 3D vision technology in the above embodiments, the present invention also proposes a first-time birth character detection and recognition system based on 3D vision technology.

[0062] like Figure 5 As shown, the first-time character detection and recognition system based on 3D vision technology in this embodiment of the invention includes: an acquisition module 10, a preprocessing module 20, an image processing module 30, an annotation module 40, a first training module 50, a second training module 60, and a display module 70. The system comprises the following modules: an acquisition module 10, which acquires point cloud data of the character region on the tire sidewall; a preprocessing module 20, which preprocesses the point cloud data, including filtering out outlier point cloud data and fitting and unfolding the curved point cloud of the tire sidewall into a planar point cloud; an image processing module 30, which converts the preprocessed point cloud data into a depth image; an annotation module 40, which annotates the depth image and generates a file corresponding to the annotation information, wherein the file includes coordinate data samples of the character region and character data samples within the character region; a first training module 50, which trains a character detection network based on the coordinate data samples of the character region to obtain a character detection model; a second training module 60, which trains a character recognition network based on the character data samples of the character region to obtain a character recognition model; and a display module 70, which detects and recognizes the character region on the tire sidewall using the character detection model and the character recognition model, and displays the detection results of the character detection model and the recognition results of the character recognition model in the original image.

[0063] In one embodiment of the present invention, a 3D line-scanning laser camera can be placed on a camera mount facing the side of the tire, and the first tire character detection device 11 can be activated to acquire point cloud data of the character area on the side of the tire, wherein the tire can be fixed on the tire mount.

[0064] In one embodiment of the present invention, such as Figure 6 and Figure 7 As shown, the first tire character detection device 11 may include a line structured light three-dimensional measuring device 111, a tire fixing and rotating device 112, and a base 113. The line structured light three-dimensional measuring device 111 consists of a column 1111 and a line structured light three-dimensional measuring device 1112. The column 1111 stands on the tire base 113 and is fixed to the tire base 113 at its bottom with hexagonal bolts. The line structured light measuring device 1112 is fixed to the column 1111 with screws.

[0065] In one embodiment of the present invention, such as Figure 6 and Figure 7As shown, the tire fixing and rotating device 112 consists of a tire fixing bracket 1121, a stepper motor 1122, a roller 1123, and a tire 1124. The tire fixing bracket 1121 is fixed to the tire base 113 with hexagonal bolts. One end of the rotating shaft in the middle of the tire is fixed to the tire's wheel center cover, and the other end passes through the shaft hole of the bracket and is fixed to the bracket by a bearing and a flange at the top of the bracket's shaft hole. The stepper motor 1122 is mounted on the tire fixing bracket 1121. When the start button is pressed, the stepper motor 1122 rotates at the speed set by the system, and the roller 1123 fixed to one end of the motor also rotates accordingly. The friction between the inner wall of the tire and the roller drives the tire to rotate.

[0066] In one embodiment of the present invention, since the character region in the acquired tire point cloud data is mainly concentrated within a fixed range along the y-axis, a point cloud pass-through filtering algorithm can be used to process the point cloud data. Specifically, the values ​​of parameters Ymin and Ymax can be set to filter out point cloud data outside the range of (Ymin, Ymax). A point cloud statistical filtering algorithm can be used to calculate the average distance between each point in the point cloud data and its ten neighboring points. Points with an average distance greater than one standard deviation of the standard distance can be considered outliers and removed from the point cloud data.

[0067] In one embodiment of the present invention, a spatial cylindrical surface of the tire surface point cloud can be fitted first, and each point can be unfolded into a planar point cloud along the X-axis. Specifically, firstly, due to the geometric properties of the cylindrical surface, it is known that the cylindrical surface is a collection of point cloud data equidistant from the center. Therefore, seven parameters of the cylindrical surface can be determined, namely the coordinates of a point on the axis (…). , , ), the axis direction vector (a, b, c), and the radius R of the cylinder. Wherein, the coordinates of a point on the axis (a, b, c) , , The coordinates of all points are the average of the coordinates of all points, and the direction vector is a unit vector pointing in the positive direction. Furthermore, it is stipulated that a > 0; if a = 0, then b > 0; if a = 0 and b = 0, then c > 0, meaning that a, b, and c cannot all be 0 simultaneously. Secondly, based on the error equation, relatively close initial values ​​can be assigned to the fitting parameters, where the error equation is:

[0068]

[0069] in,( , , Let be the coordinates of any point on the axis, and i < R. Linearizing the error equation, we have: ,in,

[0070]

[0071]

[0072]

[0073] in, For any observation point ( , , The perpendicular distance from the axis to the actual radius is then used. The solution can then be obtained through least-squares iterative calculation using the indirect difference of squares method with additional constraints. , , The system uses seven parameters, a, b, c, and R, to obtain a fitted cylindrical surface of the tire sidewall point cloud. Additional constraints are as follows:

[0074]

[0075] .

[0076] Finally, we can set the coordinates of any point A in the point cloud data on the tire surface as ( , , ), take the coordinates of a point B on the fitted cylindrical surface that is coplanar with point A as ( , , Expand point A as point, The coordinates of the point are ( , , ).in, θ is the angle between line AB and the Z-axis. The arc length can be calculated using the arc length formula. .like < ,but ;like > ,but .in, R is the radius of the fitted cylindrical surface. Let be the distance between point A and point B. By repeating the above steps to calculate all points in the point cloud data, the point cloud surface can be unfolded into a planar point cloud.

[0077] In one embodiment of the present invention, the preprocessed point cloud data can first be converted into a grayscale image. Specifically, a point M in the point cloud data can be selected. , , The value after grayscale conversion of point M is ( , (grey). Among them, Point , Then, by analyzing the height difference between the tire character and non-character regions along the Z-axis, the height information on the Z-axis is converted into depth information. Specifically, the Z-values ​​of all point cloud data are traversed to obtain the height information along the Z-axis in the point cloud data. and , Finally, all point cloud data are converted according to the above steps to obtain a depth image of the point cloud data for the tire character.

[0078] In one embodiment of the present invention, a rectangular ROI region can be set to select the character region. At the same time, a file corresponding to the annotation information can be generated. The file can be a txt file, and the file can store the path address of the depth image, the coordinates of the four vertices of each rectangular ROI region, and the character information selected by each ROI region.

[0079] In one embodiment of the present invention, a character detection network can be trained using a ResNet50 network based on the DB algorithm. Specifically, the training may include: passing a depth image through a ResNet50 network with a feature pyramid structure, generating a feature layer by upsampling, predicting a dynamic threshold probability map and a text probability map through the feature layer, forming a DB binary map through the dynamic threshold probability map and the text probability map, and generating a text box based on the DB binary map and the label.

[0080] Specifically, to acquire receptive fields of different sizes and improve the detection accuracy for different feature font regions, binarization can be inserted into the segmentation network for joint optimization. This allows the network to adaptively predict the threshold for each pixel in the image, thus completely distinguishing foreground and background pixels. First, the input image can be processed through a ResNet50 backbone network with convolutional kernels of different sizes, outputting feature maps of 1 / 2, 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image size. An FPN structure is then added to construct a feature pyramid structure. For example, the 1 / 32 feature map can be upsampled to 1 / 16 of its original size and fused with the 1 / 16 feature map to obtain a new feature map of 1 / 16 of the original image size. This results in four feature maps of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image size. The four feature maps of sizes 1 / 4, 1 / 8, 1 / 16, and 1 / 32 can be upsampled to 1 / 4 size respectively, and then concatenated with the data from the four feature maps through a concat layer to generate a feature layer.

[0081] Furthermore, the aforementioned feature layer is first processed through 1 A convolutional layer of size 1 is used, compressing the channels to 1 / 4 of the input. After passing through Batch Normalization (BN) and ReLU, the resulting feature map has a shape of (batch, 64, 1 / 4W, 1 / 4H). This feature map is then deconvolved with a kernel of size (2, 2), resulting in a feature map with a shape of (batch, 256, 1 / 2W, 1 / 2H), which is half the size of the original image. Another deconvolution operation is performed, outputting a feature map with 1 channel, now with a shape of (batch, W, H), the same size as the original image. This is then passed through a sigmoid activation function, outputting a text probability map P and a dynamic threshold value for the threshold probability map. The threshold value is then used in the binarization formula to obtain the binarized threshold probability map T. The binarization formula is:

[0082]

[0083] in, This is an approximate binary image, where k is the magnification factor, typically taken as 50, and P and T are the calculated results.

[0084] Finally, P and T can be jointly calculated using the differentiable binarization formula to obtain the final DB binarized probability map, and a text box can be generated based on the DB binarized probability map and the label.

[0085] In one embodiment of the present invention, the character recognition network may be a CRNN network based on the CTC algorithm. The CRNN network includes a convolutional layer, a recurrent layer and a transcriptional layer, wherein the recurrent layer and the transcriptional layer are connected.

[0086] Specifically, the convolutional layer extracts feature sequences from the input image. A ResNet34 network can be used to extract these features before feeding them into the CRNN text recognition algorithm's network structure. The recurrent layer predicts the label distribution of the feature sequences obtained from the convolutional layer. A bidirectional LSTM recurrent neural network can be used, where LSTM is a special type of RNN designed to address long-range dependencies. Ordinary RNNs suffer from gradient minimization or exploding, failing to capture sufficient contextual information. Therefore, the LSTM used in CRNN allows for the capture of long-range dependencies. The transcription layer transforms the label distribution obtained from the recurrent layer into the final recognition result through deduplication and integration. The CTC algorithm can be used to parse the result. CTC is a loss calculation method that introduces blank characters to address the issue of missing characters in certain positions and rapidly calculates gradients through recursion. First, a CNN extracts convolutional features from the image. Then, an LSTM further extracts sequence features from the convolutional features. Finally, CTC is introduced to address the character alignment issues during training.

[0087] In one embodiment of the present invention, the detected character region and the recognized character result can be displayed on the software interface. The output with the result below the detection and recognition threshold can be blocked, and only the result with the result above the threshold can be output.

[0088] According to an embodiment of the present invention, the first tire character detection and recognition method based on 3D vision technology acquires point cloud data of the character region on the tire sidewall through an acquisition module, and converts the point cloud data into a depth image after preprocessing by an image processing module. The depth image is labeled and coordinate data samples and character data samples are generated. The network is trained based on the coordinate data samples and character data samples to obtain a character detection and recognition model, and the character region on the tire sidewall is detected and recognized. This reduces the production cost of enterprises and improves the accuracy and efficiency of first tire character detection.

[0089] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. "A plurality of" means two or more, unless otherwise explicitly specified.

[0090] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0091] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "over," and "on top" of the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0092] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0093] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0094] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0095] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0096] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it includes one or a combination of the steps of the method embodiments.

[0097] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0098] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A first character detection and recognition method based on 3D vision technology, characterized in that, Includes the following steps: Obtain point cloud data of the character area on the tire sidewall; The point cloud data is preprocessed, including filtering out outlier point cloud data and fitting the curved point cloud of the tire sidewall and unfolding it into a planar point cloud. Convert the preprocessed point cloud data into a depth image; The depth image is annotated, and a file corresponding to the annotation information is generated, wherein the file includes coordinate data samples of the character region and character data samples within the character region; The character detection network is trained based on the coordinate data samples of the character region to obtain the character detection model; The character recognition network is trained based on character data samples from the character region to obtain a character recognition model; The character detection model and the character recognition model are used to detect and recognize the character region on the tire sidewall. The detection results of the character detection model and the recognition results of the character recognition model are then displayed in the original image. First, a spatial cylindrical surface of the tire surface point cloud is fitted, and then each point is unfolded into a planar point cloud along the X-axis. Specifically, the parameters of the cylindrical surface are first determined, which are the coordinates of a point on the axis ( , , Given the axial direction vector (a, b, c) and the radius R of the cylinder, initial values ​​are assigned to the fitting parameters based on the error equation, where the error equation is: in,( , , Let be the coordinates of any point on the axis, and i < R. Linearizing the error equation, we have: ,in, wherein, is the vertical distance from any observation point (x, y) to the axis, i.e. the actual radius, and then the least square iteration is calculated by the indirect square difference method with additional constraints , , ) to the vertical distance on the axis, i.e. the actual radius, and then the least square iteration is calculated by the indirect square difference method with additional constraints 、 、 , a, b, c and R the seven parameters, and a fitted cylindrical surface of the tire side surface point cloud is obtained, wherein the additional constraints are: Finally, let the coordinates of any point A in the point cloud data on the tire surface be ( , , ), take the coordinates of a point B on the fitted cylindrical surface that is coplanar with point A as ( , , Expand point A as point, The coordinates of the point are ( , , ),in, θ is the angle between line AB and the Z-axis. The arc length is calculated using the arc length formula. ,like < ,but ;like > ,but ,in, R is the radius of the fitted cylindrical surface. Given the distance between point A and point B, repeat the above steps to calculate all points in the point cloud data, and unfold the point cloud surface into a planar point cloud.

2. The method for first-time character detection and recognition based on 3D vision technology according to claim 1, characterized in that, The preprocessed point cloud data is converted into a depth image, specifically including: Convert the preprocessed point cloud data into a grayscale image; By analyzing the height difference between the tire character and non-character regions along the Z-axis, the height information on the Z-axis is converted into depth information. Convert the point cloud data of the tire characters into a depth image.

3. The method for first-time character detection and recognition based on 3D vision technology according to claim 2, characterized in that, The character detection network is a ResNet50 network based on the DB algorithm.

4. The method for detecting and recognizing first-time fetal characters based on 3D vision technology according to claim 3, characterized in that, The ResNet50 network is trained using the DB algorithm, specifically including: The depth image is used to generate a feature layer through an upsampling method using a ResNet50 network with a feature pyramid structure. The dynamic threshold probability map and the text probability map are obtained through the feature layer prediction. A binary DB graph is formed by combining the dynamic threshold probability graph and the text probability graph, and a text box is generated by combining the label with the binary DB graph.

5. The method for first-time character detection and recognition based on 3D vision technology according to claim 4, characterized in that, The character recognition network is a CRNN network based on the CTC algorithm. The CRNN network includes a convolutional layer, a recurrent layer, and a transcriptional layer, wherein the recurrent layer and the transcriptional layer are connected.

6. A first-time character detection and recognition system based on 3D vision technology, characterized in that, include: The acquisition module is used to acquire point cloud data of the character region on the side of the tire; The preprocessing module is used to preprocess the point cloud data, including filtering out outlier point cloud data and fitting and unfolding the curved point cloud of the tire side surface into a planar point cloud. Image processing module, which is used to convert preprocessed point cloud data into depth images; The annotation module is used to annotate the depth image and generate a file corresponding to the annotation information, wherein the file includes coordinate data samples of the character region and character data samples within the character region; The first training module is used to train the character detection network based on the coordinate data samples of the character region to obtain a character detection model. The second training module is used to train the character recognition network based on the character data samples of the character region to obtain a character recognition model. The display module is used to detect and recognize the character regions on the tire sidewall using the character detection model and the character recognition model, and to display the detection results of the character detection model and the recognition results of the character recognition model respectively in the original image. The preprocessing module first fits the spatial cylindrical surface of the tire surface point cloud, and then unfolds each point into a planar point cloud along the X-axis. Specifically, it first determines the parameters of the cylindrical surface, which are the coordinates of a point on the axis ( , , Given the axial direction vector (a, b, c) and the radius R of the cylinder, initial values ​​are assigned to the fitting parameters based on the error equation, where the error equation is: in,( , , Let be the coordinates of any point on the axis, and i < R. Linearizing the error equation, we have: ,in, in, For any observation point ( , , The perpendicular distance from the axis to the actual radius is then used to calculate the solution using the least squares iterative method with additional constraints. , , The system uses seven parameters, a, b, c, and R, to obtain a fitted cylindrical surface of the tire sidewall point cloud. Additional constraints are: Finally, let the coordinates of any point A in the point cloud data on the tire surface be ( , , ), take the coordinates of a point B on the fitted cylindrical surface that is coplanar with point A as ( , , Expand point A as point, The coordinates of the point are ( , , ),in, θ is the angle between line AB and the Z-axis. The arc length is calculated using the arc length formula. ,like < ,but ;like > ,but ,in, R is the radius of the fitted cylindrical surface. Given the distance between point A and point B, repeat the above steps to calculate all points in the point cloud data, and unfold the point cloud surface into a planar point cloud.

7. The first-time character detection and recognition system based on 3D vision technology according to claim 6, characterized in that, The preprocessed point cloud data is converted into a depth image, specifically including: Convert the preprocessed point cloud data into a grayscale image; By analyzing the height difference between the tire character and non-character regions along the Z-axis, the height information on the Z-axis is converted into depth information. Convert the point cloud data of the tire characters into a depth image.

8. The first-time character detection and recognition system based on 3D vision technology according to claim 7, characterized in that, The character detection network is a ResNet50 network based on the DB algorithm.

9. The first-time character detection and recognition system based on 3D vision technology according to claim 8, characterized in that, The ResNet50 network is trained using the DB algorithm, specifically including: The depth image is used to generate a feature layer through an upsampling method using a ResNet50 network with a feature pyramid structure. The dynamic threshold probability map and the text probability map are obtained through the feature layer prediction. A binary DB graph is formed by combining the dynamic threshold probability graph and the text probability graph, and a text box is generated by combining the label with the binary DB graph.

10. The first-time fetal character detection and recognition system based on 3D vision technology according to claim 9, characterized in that, The character recognition network is a CRNN network based on the CTC algorithm, wherein the CRNN network includes a convolutional layer, a recurrent layer and a transcriptional layer, wherein the recurrent layer and the transcriptional layer are connected.

Images