Method for generating data for industrial character recognition network training and related equipment

Abstract

The invention relates to the technical field of industrial character recognition, and provides an industrial character recognition network training data generation method and related equipment, and the method comprises the steps: obtaining a plurality of industrial character sequence images; performing surface normal estimation on each industrial character sequence image to obtain a surface normal graph of each industrial character sequence image; for each industrial character sequence image, generating a character sequence mask according to the surface normal graph of the industrial character sequence image, and extracting a plurality of character surface normal graphs from the surface normal graph of the industrial character sequence image according to the character sequence mask; matching a corresponding character label for each character surface normal graph to obtain a character normal image set; generating a final character sequence surface normal graph based on the character normal image set; and performing illumination rendering on the final character sequence surface normal graph to obtain final industrial character sequence image data. According to the method, data generation diversity can be improved, and time consumption can be reduced.

Description

Technical Field

[0001] The present application relates to the technical field of industrial character recognition, and particularly relates to a method for generating data for training an industrial character recognition network and related devices. Background Art

[0002] Industrial metal characters are usually formed on the surface of metal products by methods such as stamping or embossing, presenting as raised or sunken characters, which have the characteristics of wear resistance, corrosion resistance, and adaptability to harsh environments, and can usually maintain their shape and readability for a long time. Modern commonly used information-based product identification methods such as two-dimensional codes and barcodes may lose their readability due to surface wear, pollution, etc. under harsh working conditions such as friction, impact, and chemical corrosion. Therefore, in the industrial processing and manufacturing fields such as metal casting, manufacturers usually emboss character sequences on products to identify important information such as the batch number, specification, production date, etc. of the products. At present, many manufacturers still use the traditional method of manually reading industrial character sequences to record and manage product information. This method is easily affected by worker fatigue and negligence, and has problems such as low efficiency, high error rate, and low informatization level. Therefore, developing an efficient and accurate industrial character sequence detection and recognition system to reduce manual intervention through automated means is of great significance for improving production efficiency, enhancing the informatization level of factories, and realizing product traceability management.

[0003] In the field of optical character recognition, deep learning models are often used for the detection and recognition of character sequences. High-quality datasets are crucial for the training, validation, and testing of models. Currently, the number of publicly available industrial metal character datasets is very limited, making it difficult to meet the actual application requirements. At the same time, there are significant differences among different manufacturers in terms of character font, size, layout, marking method, etc., making it difficult for general text datasets to adapt to diverse production environments. In addition, different from the principle of conventional planar text where visibility is achieved by a distinct contrast between character color and the background, industrial characters are usually embossed or recessed on the metal surface, and visibility is achieved by the contrast between the shaded and highlighted areas under light. Secondly, the complex geometric shape of the workpiece may also form shadow occlusion on some areas of the embossed character sequence, resulting in inconsistent visibility of different characters in the same sequence. These factors make the performance of the industrial character sequence detection and recognition system more sensitive to changes in lighting conditions. This poses higher requirements for the diversity of lighting conditions covered by the dataset. In specific production applications, it is usually necessary to collect and label a large amount of data for a specific production environment and character form to ensure that the character sequence detection and recognition system has sufficient accuracy and adaptability. However, in the actual sampling process in the factory, on the one hand, the character sequence data of the same batch of products are often highly similar, with only a few characters different, making it difficult to obtain sufficient diversity in character sequence data; on the other hand, it is costly and time-consuming to obtain and label a large number of industrial character images with different orientations, lighting, and surface conditions. It can be seen that the current method for generating data for industrial character recognition network training has problems of low diversity and high time consumption in data generation. Summary of the Invention

[0004] This application provides a method for generating data for training an industrial character recognition network and related devices, which can solve the problems of low diversity and high time consumption in data generation.

[0005] In a first aspect, an embodiment of this application provides a method for generating data for training an industrial character recognition network, and the generation method includes:

[0006] Obtain a plurality of industrial character sequence images;

[0007] Perform surface normal estimation on each industrial character sequence image to obtain a surface normal map of each industrial character sequence image;

[0008] For each industrial character sequence image respectively, generate a character sequence mask according to the surface normal map of the industrial character sequence image, and extract a plurality of character surface normal maps from the surface normal map of the industrial character sequence image according to the character sequence mask; the character sequence mask is used to describe whether each pixel in the surface normal map belongs to the character region;

[0009] Match the corresponding character label for each character surface normal map to obtain a set of character normal images;

[0010] Generate a final character sequence surface normal map based on the set of character normal images;

[0011] Perform lighting rendering on the final character sequence surface normal map to obtain the final industrial character sequence image data; the final industrial character sequence image data is used to train an industrial character recognition network.

[0012] Optionally, generate a character sequence mask according to the surface normal map of the industrial character sequence image, including:

[0013] For each pixel in the surface normal map of the industrial character sequence image, calculate the normal difference between the pixel and each adjacent pixel. If all normal differences are less than the difference threshold, obtain the judgment result that the pixel belongs to the background area. If there is at least one normal difference greater than or equal to the difference threshold, obtain the judgment result that the pixel belongs to the character area, and generate a pixel mask for the pixel according to the judgment result;

[0014] Integrate all pixel masks to obtain the character sequence mask corresponding to the surface normal map.

[0015] Optionally, calculate the normal difference between the pixel and each adjacent pixel, including:

[0016] Through the formula:

[0017]

[0018] Calculate the normal difference D(p i ,p j ) between the i-th pixel and the j-th adjacent pixel;

[0019] where, n i represents the normal vector of the i-th pixel p i , n j represents the normal vector of the j-th adjacent pixel p j , n i,k represents the component of the normal vector of the i-th pixel in the k-th dimension, n j,k represents the component of the normal vector of the j-th pixel in the k-th dimension, j ∈ {1, 2,..., J}, and J represents the number of the last adjacent pixel of the i-th pixel;

[0020] Generate a pixel mask for the pixel according to the judgment result, including:

[0021] Through the formula:

[0022]

[0023] Generate the pixel mask M for the i-th pixel seq (p i )

[0024] Optionally, according to the character sequence mask, extract multiple character surface normal maps from the surface normal map of the industrial character sequence image, including:

[0025] Generate region labels for multiple pixels in the surface normal map according to the character sequence mask; the region labels are used to describe the character regions to which the pixels belong, and each character region corresponds to at least one region label;

[0026] For each character region respectively, map the region labels of all pixels in the character region to a unified root label;

[0027] Filter all character regions according to the root labels of all character regions to obtain multiple target character regions;

[0028] Extract the image regions corresponding to each target character region from the surface normal map to obtain multiple character surface normal maps.

[0029] Optionally, generate region labels for multiple pixels in the surface normal map according to the character sequence mask, including:

[0030] For each pixel in the surface normal map whose pixel mask is equal to 1, perform the following steps:

[0031] When both the first adjacent pixel and the second adjacent pixel of the pixel have region labels, and the region label of the first adjacent pixel is not equal to the region label of the second adjacent pixel, then use the smaller of the region label of the first adjacent pixel and the region label of the second adjacent pixel as the region label of the pixel, and record that there is an equivalence relationship between the region label of the first adjacent pixel and the region label of the second adjacent pixel; the equivalence relationship is used to describe that the two region labels correspond to the same character region;

[0032] When both the first adjacent pixel and the second adjacent pixel of the pixel have region labels, and the region label of the first adjacent pixel is equal to the region label of the second adjacent pixel, then use the region label as the label of the pixel;

[0033] When both the first adjacent pixel and the second adjacent pixel of the pixel do not have region labels, through the formula:

[0034] L(p i ) = C′

[0035] C′ ← C + 1

[0036] Assign a new region label L(p i ) to the pixel;

[0037] Among them, C represents the maximum region label value existing in all current pixels, and C' represents the updated region label value.

[0038] Optionally, generating a final character sequence surface normal map based on a set of character normal maps includes:

[0039] Selecting multiple target character labels from all character labels of the set of character normal maps;

[0040] For each target character label respectively, extracting a target character surface normal map from all character surface normal maps corresponding to the target character label;

[0041] Calculating the position information of each target character surface normal map, and randomly floating the position information of each target character surface normal map to obtain the final position information of each target character surface normal map;

[0042] Arranging all target character surface normal maps according to all the final position information to obtain the final character sequence surface normal map.

[0043] Optionally, calculating the position information of each target character surface normal map includes:

[0044] Through the formula:

[0045] x r = x0 + (r - 1)·d

[0046] y r = y0

[0047] θ r = θ0

[0048] Calculating the position information {x r , y r , θ r} of the r-th target character surface normal map;

[0049] Among them, x r represents the horizontal position of the center of the r-th target character surface normal map, y r represents the vertical position of the center of the r-th target character surface normal map, θ r represents the reference rotation angle of the r-th target character surface normal map with respect to the vertical direction, x0 represents the initial horizontal position, y0 represents the initial vertical position, and θ0 represents the initial reference rotation angle;

[0050] Randomly floating the position information of each target character surface normal map to obtain the final position information of each target character surface normal map includes:

[0051] Through the formula:

[0052] x' r = x r + Δx r

[0053] y' r = y r + Δy r

[0054] θ' r = θ r + Δθ r

[0055] Δx r ∈ (-Δx max , Δx max ), Δy r ∈ (-Δy max , Δy max ), Δθ r ∈ (-Δθ max , Δθ max )

[0056] Calculate the final position information {x' r , y' r , θ' r} of the r-th target character surface normal map;

[0057] where x' r represents the final horizontal position of the center of the r-th target character surface normal map, y' r represents the final vertical position of the center of the r-th target character surface normal map, θ' r represents the final reference rotation angle of the r-th target character surface normal map with respect to the vertical direction, Δx r represents the horizontal random floating value, Δy r represents the vertical random floating value, Δθ r represents the random floating value of the rotation angle, Δx max represents the maximum horizontal floating range, Δy max represents the maximum vertical floating range, Δθ max represents the maximum floating range of the rotation angle.

[0058] Optionally, perform lighting rendering on the final character sequence surface normal map to obtain the final industrial character image data, including:

[0059] Calculate the light source position and light source intensity of the point light source;

[0060] Based on the light source intensity and light source position, perform lighting rendering on each pixel in the final character sequence surface normal map to obtain the lighting result of each pixel;

[0061] Integrate the lighting results of all pixels in the final character sequence surface normal map to obtain the final industrial character sequence image data.

[0062] Optionally, calculate the light source position and light source intensity of the point light source, including:

[0063] Through the formula:

[0064] ΔL = (Δx, Δy, Δz)

[0065] L = L0 + ΔL = (x0 + Δx, y0 + Δy, z0 + Δz)

[0066] Calculate the light source position L;

[0067] Among them, ΔL represents the random offset of the light source position, Δx represents the horizontal random offset, Δy represents the vertical random offset, Δz represents the vertical random offset, L0 represents the initial light source position, x0 represents the initial horizontal coordinate, y0 represents the initial vertical coordinate, and z0 represents the initial vertical coordinate;

[0068] Through the formula:

[0069] I l = I0 + ΔI

[0070] Calculate the light source intensity I l ;

[0071] Among them, I0 represents the initial light source intensity, and ΔI represents the random change in the light source intensity;

[0072] Based on the light source intensity and light source position, perform lighting rendering on each pixel in the final character sequence surface normal map to obtain the lighting result of each pixel, including:

[0073] Through the formula:

[0074] I lit (p) = I Ambient (p) + I Diffuse (p) + I Specular (p)

[0075] Calculate the lighting result I lit (p) of pixel p in the final character sequence surface normal map;

[0076] Among them, I Ambient (p) represents the ambient reflected light intensity of pixel p, I Diffuse (p) represents the diffuse reflected light intensity of pixel p, I Specular (p) represents the specular reflected light intensity of pixel p:

[0077] I Ambient (p) = I a·k a

[0078] I Diffuse (p) = I c ·k d ·max(0, N·L)

[0079] I Specular (p) = I c ·k s ·(max(0, R·V)) n

[0080] Wherein, I a represents the intensity of the ambient light source, k a represents the ambient light reflection coefficient of the object material, k d represents the diffuse reflection coefficient of the object material, N represents the normal vector, L represents the direction from pixel p to the light source position, k s represents the specular reflection coefficient of the object material, R represents the unit vector of the reflected light direction, V represents the unit vector of the line-of-sight direction, n represents the specular glossiness coefficient, I c represents the light intensity actually reaching the character surface:

[0081]

[0082] Wherein, both k1 and k2 represent attenuation coefficients, and d represents the distance from the point light source to pixel p.

[0083] In a second aspect, an apparatus for generating data for training an industrial character recognition network provided by an embodiment of the present application includes:

[0084] An acquisition module, configured to acquire a plurality of industrial character sequence images;

[0085] A surface normal estimation module, configured to perform surface normal estimation on each industrial character sequence image to obtain a surface normal map of each industrial character sequence image;

[0086] An extraction module, configured to respectively generate a character sequence mask for each industrial character sequence image according to the surface normal map of the industrial character sequence image, and extract a plurality of character surface normal maps from the surface normal map of the industrial character sequence image according to the character sequence mask; the character sequence mask is used to describe whether each pixel in the surface normal map belongs to the character region;

[0087] A matching module, configured to match a corresponding character label to each character surface normal map to obtain a character normal image set;

[0088] A generation module, configured to generate a final character sequence surface normal map based on the character normal image set;

[0089] A lighting rendering module, which is used to perform lighting rendering on the surface normal map of the final character sequence to obtain the final industrial character sequence image data; the final industrial character sequence image data is used to train an industrial character recognition network.

[0090] In a third aspect, an embodiment of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the above-mentioned method for generating data for training an industrial character recognition network is implemented.

[0091] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned method for generating data for training an industrial character recognition network is implemented.

[0092] The above solution of the present application has the following beneficial effects:

[0093] In the embodiment of the present application, by obtaining a plurality of industrial character sequence images, then performing surface normal estimation on each industrial character sequence image to obtain the surface normal map of each industrial character sequence image, and then respectively for each industrial character sequence image, according to the surface normal map of the industrial character sequence image, generating a character sequence mask, and according to the character sequence mask, extracting a plurality of character surface normal maps from the surface normal map of the industrial character sequence image, then matching a corresponding character label for each character surface normal map to obtain a character normal image set, and finally generating a final character sequence surface normal map based on the character normal image set, and performing lighting rendering on the final character sequence surface normal map to obtain the final industrial character sequence image data. Among them, generating the final industrial character sequence image data based on the character normal image set can simulate different character morphologies, lighting conditions, and surface characteristics, provide representative and diverse data for the training of the industrial character recognition network. At the same time, generating training data does not require manual operation, avoiding human errors and reducing the time-consuming for obtaining real data.

[0094] Other beneficial effects of the present application will be described in detail in the subsequent specific implementation part. Description of the Drawings

[0095] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the embodiments or the description of the prior art. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0096] Figure 1Flowchart of a method for generating data for training an industrial character recognition network provided by an embodiment of the present application;

[0097] Figure 2 Structural schematic diagram of a device for generating data for training an industrial character recognition network provided by an embodiment of the present application;

[0098] Figure 3 Structural schematic diagram of a terminal device provided by an embodiment of the present application. Detailed implementation manners

[0099] In the following description, for the purpose of illustration rather than limitation, specific details such as specific system architectures and technologies are presented to provide a thorough understanding of the embodiments of the present application. However, those skilled in the art should clearly understand that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from interfering with the description of the present application.

[0100] It should be understood that when used in the specification of the present application and the appended claims, the term "comprising" indicates the presence of the described features, wholes, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0101] It should also be understood that the term "and / or" used in the specification of the present application and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0102] As used in the specification of the present application and the appended claims, the term "if" can be interpreted as "when", "once", "in response to determining", or "in response to detecting" depending on the context. Similarly, the phrase "if determined" or "if detecting [the described condition or event]" can be interpreted as meaning "once determined", "in response to determining", "once detecting [the described condition or event]", or "in response to detecting [the described condition or event]" depending on the context.

[0103] In addition, in the description of the specification of the present application and the appended claims, the terms "first", "second", "third", etc. are only used for distinguishing descriptions and cannot be understood as indicating or implying relative importance.

[0104] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that a specific feature, structure, or characteristic described in connection with that embodiment is included in one or more embodiments of this application. Thus, statements such as "in one embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments", etc. that appear in different places in this specification do not necessarily all refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized. The terms "comprising", "including", "having" and their variants all mean "including but not limited to", unless otherwise specifically emphasized.

[0105] Aiming at the problems of low diversity and high time consumption in the generation of existing data, the embodiments of this application provide a method for generating data for training an industrial character recognition network. The generation method obtains multiple industrial character sequence images, then performs surface normal estimation on each industrial character sequence image to obtain the surface normal map of each industrial character sequence image. Then, for each industrial character sequence image respectively, according to the surface normal map of the industrial character sequence image, a character sequence mask is generated, and according to the character sequence mask, multiple character surface normal maps are extracted from the surface normal map of the industrial character sequence image. Then, a corresponding character label is matched for each character surface normal map to obtain a character normal image set. Finally, a final character sequence surface normal map is generated based on the character normal image set, and the final character sequence surface normal map is subjected to lighting rendering to obtain the final industrial character sequence image data. Among them, generating the final industrial character sequence image data based on the character normal image set can simulate different character forms, lighting conditions, and surface characteristics, providing representative and diverse data for the training of the industrial character recognition network. At the same time, generating training data does not require manual operation, avoiding human errors and reducing the time consumption for obtaining real data.

[0106] Next, an exemplary description will be given of the method for generating data for training an industrial character recognition network provided in this application.

[0107] As Figure 1 shown, the method for generating data for training an industrial character recognition network provided in this application includes the following steps:

[0108] Step 11, obtain multiple industrial character sequence images.

[0109] The industrial character sequence image is image data including industrial character sequences in the actual industrial production environment. For example, in the industry of steel production, the industrial character sequence image is an image of character sequences such as the numbers and production dates imprinted on the produced steel.

[0110] In some embodiments of the present application, an industrial character sequence image can be obtained through devices such as cameras. The industrial character sequence image needs to have uniform illumination and a high resolution, and preferably only includes the local area where the industrial character sequence is imprinted on the workpiece, avoiding large areas including the workpiece background area, and ensuring that the outlines and features of the characters are clearly visible.

[0111] Step 12: Perform surface normal estimation on each industrial character sequence image to obtain the surface normal map of each industrial character sequence image.

[0112] The above surface normal map is used to describe the normal vector of each pixel in the industrial character sequence image.

[0113] In some embodiments of the present application, an existing normal estimation algorithm (such as a surface normal estimation method based on image gradients) can be used to perform surface normal estimation on each industrial character sequence image to obtain the surface normal map of each industrial character sequence image.

[0114] Step 13: For each industrial character sequence image, generate a character sequence mask according to the surface normal map of the industrial character sequence image, and extract a plurality of character surface normal maps from the surface normal map of the industrial character sequence image according to the character sequence mask.

[0115] The above character sequence mask is used to describe whether each pixel in the surface normal map belongs to the character area. An industrial character sequence image is an image used to represent industrial characters, which includes a character area describing character information and a background area without character information. For example, in an industrial character sequence image of the production date of steel, the area composed of all pixels corresponding to the production date is the character area, and the area composed of other pixels (such as pixels corresponding to steel) other than these pixels is the background area.

[0116] In some embodiments of the present application, the steps of generating a character sequence mask according to the surface normal map of the industrial character sequence image and extracting a plurality of character surface normal maps from the surface normal map of the industrial character sequence image according to the character sequence mask include:

[0117] First step: For each pixel in the surface normal map of the industrial character sequence image, calculate the normal difference between the pixel and each adjacent pixel. If all normal differences are less than the difference threshold, a judgment result that the pixel belongs to the background area is obtained. If there is at least one normal difference greater than or equal to the difference threshold, a judgment result that the pixel belongs to the character area is obtained, and a pixel mask of the pixel is generated according to the judgment result.

[0118] Specifically, through the formula:

[0119]

[0120] Calculate the normal difference D(p i , p j ) between the i-th pixel and the j-th adjacent pixel.

[0121] Where n i represents the normal vector of the i-th pixel p i , n j represents the normal vector of the j-th adjacent pixel p j , n i,k represents the component of the normal vector of the i-th pixel in the k-th dimension, and n j,k represents the component of the normal vector of the j-th pixel in the k-th dimension, where j ∈ {1, 2,..., J}, and J represents the number of the last adjacent pixel of the i-th pixel.

[0122] Generate a pixel mask for the pixel according to the judgment result, including:

[0123] Through the formula:

[0124]

[0125] Generate the pixel mask M seq (p i ) of the i-th pixel.

[0126] In the second step, integrate all the pixel masks to obtain the character sequence mask corresponding to the surface normal map.

[0127] Specifically, integrate all the pixel masks into one piece of data to obtain the character sequence mask corresponding to the surface normal map.

[0128] In the third step, generate region labels for multiple pixels in the surface normal map according to the character sequence mask.

[0129] The above region labels are used to describe the character regions to which the pixels belong, and each character region corresponds to at least one region label. In some embodiments of the present application, the region label can be a numerical value, with an initial value of 0. If the numerical value is greater than 0, the pixel has a corresponding character region; if the numerical value is equal to 0, the pixel belongs to the background region; if the pixel does not have a region label, it means that the pixel has not been processed yet.

[0130] Specifically, for each pixel with a pixel mask equal to 1 in the surface normal map, perform the following steps:

[0131] When both the first adjacent pixel and the second adjacent pixel of a pixel have region labels and the region label of the first adjacent pixel is not equal to the region label of the second adjacent pixel, then the smaller of the region label of the first adjacent pixel and the region label of the second adjacent pixel is used as the region label of the pixel, and it is recorded that there is an equivalence relationship between the region label of the first adjacent pixel and the region label of the second adjacent pixel; the equivalence relationship is used to describe that two region labels correspond to the same character region.

[0132] When both the first adjacent pixel and the second adjacent pixel of a pixel have region labels and the region label of the first adjacent pixel is equal to the region label of the second adjacent pixel, then this region label is used as the label of the pixel.

[0133] When neither the first adjacent pixel nor the second adjacent pixel of a pixel has a region label, through the formula:

[0134] L(p i ) = C′

[0135] C′ ← C + 1

[0136] A new region label L(p i ) is assigned to the pixel;

[0137] where C represents the maximum region label value existing among all current pixels, and C′ represents the updated region label value.

[0138] It should be noted that when one of the first adjacent pixel and the second adjacent pixel of a pixel has a region label, this region label is used as the region label of the current pixel; for a pixel with a pixel mask equal to 0, its region label is directly set to the initial value, and the initial value of the region label is 0. The first adjacent pixel is usually the upper pixel, and the second adjacent pixel is usually the left pixel. For a pixel without an upper pixel or a left pixel, it is considered that its upper pixel or left pixel does not have a region label.

[0139] Exemplarily, for a total of 20 pixels, among which 5 pixels have a region label of 3, 14 pixels have a region label of 4, and 1 pixel has a region label of 5, and there is an equivalence relationship between region label 3 and region label 4, indicating that the 5 pixels corresponding to region label 3 and the 14 pixels corresponding to region label 4 both belong to the same character region.

[0140] In the fourth step, for each character region, the region labels of all pixels in the character region are mapped to a unified root label.

[0141] Exemplarily, the Union-Find algorithm can be utilized to determine the root label of each character region, and then each pixel is traversed to update the region label of the pixel to the root label of the corresponding character region to ensure that each character region has only one unique label. Specifically, through the formula:

[0142]

[0143] the region label of the pixel is updated to obtain the updated region label L(p' i ).

[0144] Among them, L(p i ) represents the region label of the pixel, φ represents the mapping function, and φ(L(p i )) represents the root label of the character region corresponding to the pixel.

[0145] Step 5: Filter all character regions according to the root labels of all character regions to obtain multiple target character regions.

[0146] In some embodiments of the present application, the process of filtering all character regions according to the root labels of all character regions includes:

[0147] (1) Filter character regions based on the region area

[0148] Calculate the area of each character region in the input image. By counting the number of pixels corresponding to the root label of each character region, the actual area of the character region is calculated. Then, set a minimum region area threshold. If the area of a certain character region is less than the threshold, it is considered that the character region is a noise region and is excluded. Only the character regions with an area greater than or equal to the threshold are retained.

[0149] (2) Filter character regions based on the aspect ratio of the minimum bounding rectangle

[0150] Calculate the aspect ratio of the minimum bounding rectangle of each character region according to all pixels corresponding to the root label of each character region. According to the actual application scenario, set a reasonable aspect ratio range. If the aspect ratio of the minimum bounding rectangle of a certain character region is not within the set range, it is considered that the shape of the character region does not conform to the basic shape characteristics of the character and is excluded. Only the character regions with an aspect ratio meeting the requirements are retained.

[0151] (3) Filter character regions based on sparsity

[0152] Calculate the area of the minimum bounding rectangle for each character region based on all the pixels corresponding to the root tag of each character region, and compare it with the actual number of pixels in the character region to obtain the sparsity of the character region. Specifically, the ratio between the area of the character region and the area of the minimum bounding rectangle is called the area occupancy ratio. If the area occupancy ratio of a certain character region is less than the set threshold, it indicates that the character region is too sparse, and then this character region is considered not to be a valid character region and is excluded. Only the character regions with the area occupancy ratio meeting the requirements are retained.

[0153] Take the character regions retained after the above three types of filtering as the target character regions.

[0154] Step 6: Extract the image region corresponding to each target character region from the surface normal map to obtain multiple character surface normal maps.

[0155] Specifically, for each target character region, perform a per-pixel masking operation to obtain the corresponding character surface normal map. The calculation formula is as follows:

[0156]

[0157] where N i (p) represents the normal value at pixel p in the i-th character surface normal map, N(p) represents the normal value at pixel p in the surface normal map, and M i (p) represents the mask of pixel p for the i-th character region. If it is greater than 0, it means that this pixel belongs to the i-th character region. If it is equal to 0, it means that this pixel belongs to the background region. If M i (p)>0, then keep the original value, otherwise set it to the background value (such as 0 or other invalid values).

[0158] Step 14: Match a corresponding character label for each character surface normal map to obtain a character normal image set.

[0159] Specifically, identify each character surface normal map to ensure that each character surface normal map corresponds to a specific character label (optical character recognition technology, etc. can be used to identify the character surface normal map to obtain the character label corresponding to the character surface normal map. The character label is used to describe the character represented by the character surface normal map. For example, if the character represented in the character surface normal map is the character "year", then the corresponding character label is "year"), and then construct a character set Character set contains all the character labels. For each character label construct a set of character surface normal maps under this character label This set contains all the valid character surface normal maps corresponding to this character label. The character normal image set N CIt is a set of normal images for all character categories, that is, the stored images are all of single characters.

[0160] Step 15: Generate the final character sequence surface normal map based on the character normal image set.

[0161] In some embodiments of the present application, the step of generating the final character sequence surface normal map based on the character normal image set is specifically as follows:

[0162] First step: Select multiple target character labels from all the character labels in the character normal image set.

[0163] Exemplarily, multiple target character labels can be randomly selected from all the character labels.

[0164] Second step: For each target character label, extract a target character surface normal map from all the character surface normal maps corresponding to the target character label.

[0165] Exemplarily, a target character surface normal map can be randomly extracted from all the character surface normal maps corresponding to the target character label.

[0166] Third step: Calculate the position information of each target character surface normal map, and randomly float the position information of each target character surface normal map to obtain the final position information of each target character surface normal map.

[0167] Specifically, through the formula:

[0168] x r = x0 + (r - 1)·d

[0169] y r = y0

[0170] θ r = θ0

[0171] Calculate the position information {x r , y r , θ r} of the r-th target character surface normal map;

[0172] Among them, x r represents the horizontal position of the center of the r-th target character surface normal map, y r represents the vertical position of the center of the r-th target character surface normal map, θ r represents the reference rotation angle of the r-th target character surface normal map with respect to the vertical direction, x0 represents the initial horizontal position, y0 represents the initial vertical position, and θ0 represents the initial reference rotation angle.

[0173] Through the formula:

[0174] x' r = x r + Δx r

[0175] y' r = y r + Δy r

[0176] θ' r = θ r + Δθ r

[0177] Δx r ∈ (-Δx max , Δx max ), Δy r ∈ (-Δy max , Δy max ), Δθ r ∈ (-Δθ max , Δθ max )

[0178] Calculate the final position information {x' r , y' r , θ' r} of the r-th target character surface normal map;

[0179] Among them, x' r represents the final horizontal position of the center of the r-th target character surface normal map, y' r represents the final vertical position of the center of the r-th target character surface normal map, θ' r represents the final reference rotation angle between the r-th target character surface normal map and the vertical direction, Δx r represents the horizontal random floating value, Δy r represents the vertical random floating value, Δθ r represents the random floating value of the rotation angle, Δx max represents the maximum horizontal floating range, Δy max represents the maximum vertical floating range, Δθ max represents the maximum floating range of the rotation angle.

[0180] Step 4: Arrange all the target character surface normal maps according to all the final position information to obtain the final character sequence surface normal map.

[0181] Step 16: Perform lighting rendering on the final character sequence surface normal map to obtain the final industrial character sequence image data.

[0182] The above-mentioned final industrial character sequence image data is used to train the industrial character recognition network, and the industrial character recognition network can be a convolutional neural network, etc.

[0183] First, calculate the light source position and light source intensity of the point light source.

[0184] Specifically, through the formula:

[0185] ΔL = (Δx, Δy, Δz)

[0186] L = L0 + ΔL = (x0 + Δx, y0 + Δy, z0 + Δz)

[0187] Calculate the light source position L;

[0188] Among them, ΔL represents the random offset of the light source position, Δx represents the horizontal random offset, Δy represents the vertical random offset, Δz represents the vertical random offset, L0 represents the initial light source position, x0 represents the initial horizontal coordinate, y0 represents the initial vertical coordinate, and z0 represents the initial vertical coordinate.

[0189] Second, perform lighting rendering on each pixel in the surface normal map of the final character sequence based on the light source intensity and light source position to obtain the lighting result of each pixel.

[0190] Specifically, through the formula:

[0191] I l = I0 + ΔI

[0192] Calculate the light source intensity I l ;

[0193] Among them, I0 represents the initial light source intensity, and ΔI represents the random change in the light source intensity.

[0194] Third, integrate the lighting results of all pixels in the surface normal map of the final character sequence to obtain the final industrial character sequence image data.

[0195] Specifically, through the formula:

[0196] I lit (p) = I Ambient (p) + I Diffuse (p) + I Specular (p)

[0197] Calculate the lighting result I lit (p) of the pixel p in the surface normal map of the final character sequence;

[0198] Among them, I Ambient (p) represents the ambient reflected light intensity of the pixel p, I Diffuse (p) represents the diffuse reflected light intensity of the pixel p, I Specular (p) represents the specular reflected light intensity of the pixel p:

[0199] I Ambient (p) = I a ·k a

[0200] I Diffuse (p) = I c ·k d ·max(0, N·L)

[0201] I Specular (p) = I c ·k s ·(max(0, R·V)) n

[0202] Wherein, I a represents the intensity of the ambient light source, k a represents the ambient light reflection coefficient of the object material, k d represents the diffuse reflection coefficient of the object material, N represents the normal vector, L represents the direction from pixel p to the light source position, k s represents the specular reflection coefficient of the object material, R represents the unit vector of the reflected light direction, V represents the unit vector of the line-of-sight direction, n represents the specular glossiness coefficient, I c represents the light intensity actually reaching the character surface:

[0203]

[0204] Wherein, both k1 and k2 represent attenuation coefficients, and d represents the distance from the point light source to pixel p.

[0205] It should be noted that the above coordinates are all in a three-dimensional coordinate system with the vertex of the normal map of the final character sequence surface as the origin, the pixel row direction of the normal map of the final character sequence surface as the horizontal axis, the pixel column direction as the vertical axis, and the direction perpendicular to the normal map of the final character sequence surface as the vertical axis. By repeating this step, arbitrary permutations and combinations of characters can be performed to generate diverse sequence data. After obtaining the final industrial character sequence image data, the final industrial character sequence image data is used as the training data for the industrial character recognition network to train the industrial character recognition network, so as to improve the model performance of the industrial character recognition network when used for actual industrial character recognition.

[0206] It is worth mentioning that generating the final industrial character sequence image data based on the character normal image set can simulate different character shapes, lighting conditions, and surface characteristics, providing representative and diverse data for the training of the industrial character recognition network. At the same time, generating training data does not require manual operation, avoiding human errors and reducing the time-consuming for obtaining real data.

[0207] In addition, the generated data can not only reduce the cost of obtaining real data, but also ensure the diversity of the data, which is of great significance for helping the model better cope with complex industrial scenarios and improving the performance of the industrial character sequence detection and recognition system.

[0208] Next, an exemplary description will be given of the apparatus for generating data for training an industrial character recognition network provided in this application.

[0209] As Figure 2 shown, an embodiment of this application provides an apparatus for generating data for training an industrial character recognition network. The apparatus 200 for generating data for training an industrial character recognition network includes:

[0210] An acquisition module 201, configured to acquire a plurality of industrial character sequence images;

[0211] A surface normal estimation module 202, configured to perform surface normal estimation on each industrial character sequence image to obtain a surface normal map of each industrial character sequence image;

[0212] An extraction module 203, configured to respectively generate a character sequence mask for each industrial character sequence image according to the surface normal map of the industrial character sequence image, and extract a plurality of character surface normal maps from the surface normal map of the industrial character sequence image according to the character sequence mask; the character sequence mask is used to describe whether each pixel in the surface normal map belongs to the character region;

[0213] A matching module 204, configured to match a corresponding character label for each character surface normal map to obtain a character normal image set;

[0214] A generation module 205, configured to generate a final character sequence surface normal map based on the character normal image set;

[0215] A lighting rendering module 206, configured to perform lighting rendering on the final character sequence surface normal map to obtain final industrial character sequence image data; the final industrial character sequence image data is used to train the industrial character recognition network.

[0216] It should be noted that for the information interaction, execution process, etc. between the above-mentioned devices / units, since they are based on the same concept as the method embodiment of this application, their specific functions and the technical effects brought about can be specifically referred to in the method embodiment part, and will not be elaborated here.

[0217] Those skilled in the art can clearly understand that, for the convenience and conciseness of description, only the above-mentioned division of each functional unit and module is used as an example. In actual applications, the above functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction and do not limit the protection scope of this application. The specific working processes of the units and modules in the above system can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0218] As Figure 3 shown, an embodiment of the present application provides a terminal device. The terminal device D10 in this embodiment includes: at least one processor D100 ( Figure 3 only one processor is shown in the figure), a memory D101, and a computer program D102 stored in the memory D101 and executable on the at least one processor D100. When the processor D100 executes the computer program D102, the steps in any of the foregoing method embodiments are implemented.

[0219] Specifically, when the processor D100 executes the computer program D102, by obtaining a plurality of industrial character sequence images, then performing surface normal estimation on each industrial character sequence image to obtain the surface normal map of each industrial character sequence image, and then respectively for each industrial character sequence image, generating a character sequence mask according to the surface normal map of the industrial character sequence image, and extracting a plurality of character surface normal maps from the surface normal map of the industrial character sequence image according to the character sequence mask, then matching a corresponding character label for each character surface normal map to obtain a character normal image set, and finally generating a final character sequence surface normal map based on the character normal image set, and performing lighting rendering on the final character sequence surface normal map to obtain the final industrial character sequence image data. Among them, generating the final industrial character sequence image data based on the character normal image set can simulate different character morphologies, lighting conditions, and surface characteristics, provide representative and diverse data for the training of the industrial character recognition network. At the same time, generating training data does not require manual operation, avoids human errors, and reduces the time-consuming for obtaining real data.

[0220] The so-called processor D100 may be a central processing unit (CPU, Central Processing Unit), and the processor D100 may also be other general-purpose processors, digital signal processors (DSP, Digital Signal Processor), application specific integrated circuits (ASIC, Application Specific Integrated Circuit), field-programmable gate arrays (FPGA, Field-Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0221] In some embodiments, the memory D101 may be an internal storage unit of the terminal device D10, such as the hard disk or memory of the terminal device D10. In some other embodiments, the memory D101 may also be an external storage device of the terminal device D10, such as a plug-in hard disk, a smart media card (SMC, SmartMedia Card), a secure digital (SD, Secure Digital) card, a flash card (Flash Card), etc. equipped on the terminal device D10. Further, the memory D101 may also include both the internal storage unit and the external storage device of the terminal device D10. The memory D101 is used to store an operating system, application programs, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program, etc. The memory D101 may also be used to temporarily store data that has been output or is to be output.

[0222] The embodiment of the present application also provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned various method embodiments can be implemented.

[0223] The embodiment of the present application provides a computer program product, and when the computer program product runs on a terminal device, the terminal device can be enabled to implement the steps in the above-mentioned various method embodiments when executed.

[0224] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, to implement all or part of the processes in the above-described embodiment methods of this application, a computer program can be used to instruct relevant hardware to complete. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above-described method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the method device / terminal device for generating industrial character recognition network training data, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal, and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk, or an optical disc, etc.

[0225] In the above embodiments, the descriptions of the various embodiments each have their own emphases. For parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0226] Those of ordinary skill in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed in this document can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0227] The above is the preferred implementation manner of this application. It should be noted that for those of ordinary skill in the art of this technology, without departing from the principle described in this application, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of this application.

Claims

1. A method for generating data for training an industrial character recognition network, characterized in that, Including: Obtaining a plurality of industrial character sequence images; Performing surface normal estimation on each of the industrial character sequence images to obtain a surface normal map for each of the industrial character sequence images; For each of the industrial character sequence images respectively, generating a character sequence mask according to the surface normal map of the industrial character sequence image, and extracting a plurality of character surface normal maps from the surface normal map of the industrial character sequence image according to the character sequence mask; the character sequence mask is used to describe whether each pixel in the surface normal map belongs to a character region; Matching a corresponding character label for each character surface normal map to obtain a character normal image set; Generating a final character sequence surface normal map based on the character normal image set; Performing lighting rendering on the final character sequence surface normal map to obtain final industrial character sequence image data; The final industrial character sequence image data is used to train an industrial character recognition network.

2. The generation method according to claim 1, wherein The generating a character sequence mask according to the surface normal map of the industrial character sequence image includes: For each pixel in the surface normal map of the industrial character sequence image respectively, calculating the normal difference between the pixel and each adjacent pixel. If all the normal differences are less than a difference threshold, a judgment result that the pixel belongs to a background region is obtained. If there is at least one normal difference greater than or equal to the difference threshold, a judgment result that the pixel belongs to a character region is obtained, and a pixel mask of the pixel is generated according to the judgment result; Integrating all the pixel masks to obtain the character sequence mask corresponding to the surface normal map.

3. The generation method according to claim 2, wherein The calculating the normal difference between the pixel and each adjacent pixel includes: By the formula: Calculate the normal difference D(p i , p j ) between the i-th pixel and the j-th adjacent pixel; where n i represents the normal vector of the i-th pixel p i and n j represents the normal vector of the j-th adjacent pixel p j and n i,k represents the component of the normal vector of the i-th pixel in the k-th dimension, and n j,k represents the component of the normal vector of the j-th pixel in the k-th dimension, where j ∈ {1, 2,..., J}, and J represents the number of the last adjacent pixel of the i-th pixel; The generating a pixel mask of the pixel according to the judgment result includes: By the formula: Generate the pixel mask M for the i-th pixel seq (p i ).

4. The generation method according to claim 3, wherein The extracting a plurality of character surface normal maps from the surface normal map of the industrial character sequence image according to the character sequence mask includes: Generating region labels for a plurality of pixels in the surface normal map according to the character sequence mask; the region labels are used to describe the character regions to which the pixels belong, and each character region corresponds to at least one region label; For each of the character regions respectively, mapping the region labels of all the pixels in the character region to a unified root label; Filtering all the character regions according to the root labels of all the character regions to obtain a plurality of target character regions; Extracting an image region corresponding to each target character region from the surface normal map to obtain a plurality of character surface normal maps.

5. The generation method according to claim 4, wherein The generating region labels for a plurality of pixels in the surface normal map according to the character sequence mask includes: For each pixel whose pixel mask in the surface normal map is equal to 1 respectively, performing the following steps: When both the first adjacent pixel and the second adjacent pixel of the pixel have region labels, and the region label of the first adjacent pixel is not equal to the region label of the second adjacent pixel, then the smaller of the region label of the first adjacent pixel and the region label of the second adjacent pixel is used as the region label of the pixel, and it is recorded that there is an equivalence relationship between the region label of the first adjacent pixel and the region label of the second adjacent pixel; the equivalence relationship is used to describe that the two region labels correspond to the same character region; When both the first adjacent pixel and the second adjacent pixel of the pixel have region labels, and the region label of the first adjacent pixel is equal to the region label of the second adjacent pixel, then the region label is used as the label of the pixel; When both the first adjacent pixel and the second adjacent pixel of the pixel do not have region labels, through the formula: L(p i ) = C' C′←C+1 Assign a new region label L(p i ) to the pixel where C represents the maximum region label value existing among all current pixels, and C' represents the updated region label value.

6. The generation method according to claim 1, wherein Generating the final character sequence surface normal map based on the character normal image set includes: Selecting a plurality of target character labels from all character labels of the character normal image set; For each of the target character labels, extracting a target character surface normal map from all character surface normal maps corresponding to the target character label; Calculating the position information of each target character surface normal map, and randomly floating the position information of each target character surface normal map to obtain the final position information of each target character surface normal map; Arranging all target character surface normal maps according to all the final position information to obtain the final character sequence surface normal map.

7. The generating method according to claim 6, wherein Calculating the position information of each target character surface normal map includes: Through the formula: x r = x0 + (r - 1)·d y r = y0 θ r = θ0 Calculate the position information {x r , y r , θ r} of the r-th target character surface normal map; where x r represents the horizontal position of the center of the surface normal map of the r-th target character, y r represents the vertical position of the center of the surface normal map of the r-th target character, θ r represents the reference rotation angle of the surface normal map of the r-th target character with respect to the vertical direction, x0 represents the initial horizontal position, y0 represents the initial vertical position, and θ0 represents the initial reference rotation angle; Randomly floating the position information of each target character surface normal map to obtain the final position information of each target character surface normal map includes: Through the formula: x′ r = x r + Δx r y' r = y r + Δy r θ′ r = θ r + Δθ r Δx r ∈(-Δx max ,Δx max ),Δy r ∈(-Δy max ,Δy max ),Δθ r ∈(-Δθ max ,Δθ max ) Calculate the final position information {x' r , y' r , θ' r} of the r-th target character surface normal map; where x' r represents the final horizontal position of the center of the surface normal map of the r-th target character, y' r represents the final vertical position of the center of the surface normal map of the r-th target character, θ' r represents the final reference rotation angle between the surface normal map of the r-th target character and the vertical direction, Δx r represents the horizontal random floating value, Δy r represents the vertical random floating value, Δθ r represents the rotation angle random floating value, Δx max represents the maximum horizontal floating range, Δy max represents the maximum vertical floating range, Δθ max represents the maximum rotation angle floating range.

8. The generation method according to claim 1, wherein Performing lighting rendering on the final character sequence surface normal map to obtain the final industrial character image data includes: Calculating the light source position and light source intensity of the point light source; Based on the light source intensity and light source position, performing lighting rendering on each pixel in the final character sequence surface normal map to obtain the lighting result of each pixel; Integrating the lighting results of all pixels in the final character sequence surface normal map to obtain the final industrial character sequence image data.

9. The generation method according to claim 8, wherein Calculating the light source position and light source intensity of the point light source includes: Through the formula: ΔL=(Δx,Δy,Δz) L=L0+ΔL=(x0+Δx,y0+Δy,z0+Δz) Calculating the light source position L; where ΔL represents the random offset of the light source position, Δx represents the horizontal random offset, Δy represents the vertical random offset, Δz represents the vertical random offset, L0 represents the initial light source position, x0 represents the initial horizontal coordinate, y0 represents the initial vertical coordinate, and z0 represents the initial vertical coordinate; Through the formula: I l = I0 + ΔI Calculate the light source intensity I l ; where I0 represents the initial light source intensity, and ΔI represents the random change in the light source intensity; Performing lighting rendering on each pixel in the surface normal map of the final character sequence based on the light source intensity and the light source position to obtain the lighting result of each pixel, including: Through the formula: I lit (p) = I Ambient (p) + I Diffuse (p) + I Specular (p) Calculate the lighting result I of the pixel p in the final character sequence surface normal map lit (p); Among them, I Ambient (p) represents the ambient reflected light intensity of the pixel p, I Diffuse (p) represents the diffuse reflected light intensity of the pixel p, I Specular (p) represents the specular reflected light intensity of the pixel p: I Ambient (p) = I a ·k a I Diffuse (p) = I c ·k d ·max(0, N·L) I Specular (p) = I c ·k s ·(max0, R·V)) n Among them, I a represents the intensity of the ambient light source, k a represents the ambient light reflection coefficient of the object material, k d represents the diffuse reflection coefficient of the object material, N represents the normal vector, L represents the direction from the pixel p to the light source position, k s represents the specular reflection coefficient of the object material, R represents the unit vector of the reflected light direction, V represents the unit vector of the line-of-sight direction, n represents the specular glossiness coefficient, I c represents the light intensity actually reaching the character surface: wherein, k1 and k2 both represent attenuation coefficients, and d represents the distance from the point light source to the pixel p.

10. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method for generating data for training an industrial character recognition network according to any one of claims 1 to 9.

Images