Instrument image generation method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses an instrument image generation method and device, electronic equipment and a storage medium, and relates to the field of computer vision and artificial intelligence, and the method comprises the steps: obtaining a real image of an instrument, recognizing font information, analyzing text display characteristics of different modes, constructing a corpus according to the font information and the text display characteristics, and generating a corpus; using a text detection algorithm to generate a background image according to the real image, designing character parameters according to a corpus, determining a text size according to the character parameters, determining a corpus and drawing parameters according to the text size and the corpus, and drawing all characters in the corpus on the background image according to the drawing parameters to obtain a text image; and performing image enhancement of different types and degrees on the text image by using a multi-modal large model to generate a plurality of instrument images. According to the invention, the image types are enriched through image enhancement, the image quality is improved, and the problems of poor diversity, uncontrollable effect and large difference with a real scene in the prior art are solved.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and artificial intelligence, and in particular to an instrument image generation method, device, electronic device and storage medium. Background Art

[0002] As an important branch of image processing and artificial intelligence, OCR (Optical Character Recognition) technology has made significant progress in recent years. OCR technology can automatically recognize text information in images and convert it into editable text format, greatly improving the processing efficiency of text data. With the rise of deep learning technology, the accuracy and generalization ability of OCR models have been continuously improved, making OCR technology widely used in various application scenarios.

[0003] Instrument text recognition is characterized by standardized but variable text fonts, complex environments, and different language systems with different character sets and grammatical rules, which requires separate training of OCR models for different languages. At the same time, even for the same language, characters in different fonts vary greatly, which requires a large amount of diverse text image data for model training. However, manual annotation and review of text data is extremely time-consuming and labor-intensive, and image data in real scenes vary greatly, making it difficult to meet the needs of model training.

[0004] In addition, the existing technology also has some shortcomings. The overall generation method based on GAN or diffusion model is limited by the selection of training data and prompt words, the diversity of generated images is poor, and the effect is uncontrollable. The general text image generation solution directly draws text on the real image, resulting in a large difference between the generated text image and the real scene, and cannot be effectively applied to the real scene.

[0005] Therefore, there is an urgent need for an instrument image generation method that can generate diverse and realistic instrument images to meet the OCR model training requirements under different language systems and fonts. Summary of the invention

[0006] The embodiments of the present invention provide a method for generating an instrument image to solve the problem that the image generated by the prior art has poor diversity and uncontrollable effects, is greatly different from the real scene, and cannot be effectively applied to the real scene. The technical solution is as follows:

[0007] According to one aspect of the present invention, a method for generating an instrument image comprises: acquiring a real image of an instrument, identifying font information in the real image, analyzing text display characteristics of the instrument in different modes, and constructing a corpus according to the font information and text display characteristics; using a text detection algorithm to generate a background image according to the real image, designing character parameters according to the corpus, and determining a text size according to the character parameters; determining a corpus and drawing parameters according to the text size and the corpus, and drawing all characters in the corpus onto the background image according to the drawing parameters to obtain a text image; and using a multimodal large model to perform different types and degrees of image enhancement on the text image to generate multiple instrument images.

[0008] In one of the embodiments, analyzing the text display characteristics of the instrument in different modes and constructing a corpus based on the font information and text display characteristics is achieved by the following steps: obtaining the text content of the instrument in different modes, performing statistical analysis on all the text content to obtain a character set and a character frequency; sorting out the text combination rules of the instrument in different modes based on the character set and the character frequency to obtain a character combination rule, and randomly generating a first corpus and a second corpus based on the character frequency; generating a third corpus that conforms to each mode of the instrument based on the character combination rule, and performing character spacing control and fusion on the first corpus, the second corpus, and the third corpus to obtain a corpus.

[0009] In one of the embodiments, generating a background image according to the real image using a text detection algorithm is achieved by the following steps: using a text detection algorithm to detect a text area in the real image, randomly generating a vertex outside the text area and randomly setting width and height values; obtaining a square centered on the vertex, a rectangle with the vertex as the upper left point, and a rectangle with the vertex as the lower right point according to the width and height values; calculating whether the rectangles have intersections with the text area, taking rectangles without intersections as background images, and splicing and fusing the background images to obtain a background image.

[0010] In one of the embodiments, the background images are stitched and fused to obtain a background image by the following steps: the aspect ratios of all non-intersecting rectangles are calculated and sorted according to the aspect ratios, and two background images whose aspect ratio difference is less than 0.1 are randomly selected; the heights of the two background images are adjusted to be the same while maintaining their respective aspect ratios, and sub-regions of equal width are selected from the right end of one background image and the left end of the other background image; weights are set for the sub-regions, and a weighted fusion algorithm is used to merge the sub-regions according to the weights to obtain a background image.

[0011] In one of the embodiments, character parameters are designed according to the corpus, and the text size is determined according to the character parameters by the following steps: selecting a font and corpus according to the corpus, determining a background image and obtaining the width and height, initializing the font size, and calculating the ratio between the width and height of the corpus and the width and height of the background image; if the ratio is lower than 2 / 3, increasing the font size, and if the ratio is higher than 2 / 3, reducing the font size to obtain the text size.

[0012] In one of the embodiments, the corpus and drawing parameters are determined according to the text size and the corpus, and all the characters in the corpus are drawn onto the background image according to the drawing parameters to obtain the text image by the following steps: obtaining the number and size of the corpus, calculating the remaining interval in combination with the size of the background image, dividing the remaining interval by the integer part of the number of characters in the corpus as a fixed interval and generating a random interval; the random interval fluctuates around the value of the fixed interval; generating a font weight for each character in the corpus, using an image processing algorithm to obtain the width of the character according to the text size, and calculating the drawing position of the character according to the fixed interval, random interval, font weight and width of the character; randomly generating a gray value according to the average gray value of the character, and drawing all the characters in the corpus onto the background image according to the text size, font weight and gray value to obtain the text image.

[0013] In one of the embodiments, using a multimodal large model to perform different types and degrees of image enhancement on the text image to generate multiple instrument images is achieved by the following steps: selecting and initializing a multimodal large model, selecting a text image and defining prompt words, inputting the text image and the corresponding prompt words into the multimodal large model to generate an image description; performing enhancement processing on the text image, and inputting each enhanced text image and the corresponding prompt words into the multimodal large model again to generate an enhanced image description; the enhancement processing includes adjusting lighting, adding shadows, and simulating occlusion; calculating the text similarity between the image description and the enhanced image description, and if the text similarity meets the set conditions, the enhanced text image is used as an instrument image.

[0014] According to one aspect of the present invention, a device for generating an instrument image comprises: a corpus construction module, which is used to obtain a real image of an instrument, identify font information in the real image, analyze text display characteristics of the instrument in different modes, and construct a corpus according to the font information and text display characteristics; a data preparation module, which is used to generate a background image according to the real image using a text detection algorithm, design character parameters according to the corpus, and determine the text size according to the character parameters; a text image generation module, which is used to determine a corpus and drawing parameters according to the text size and the corpus, and draw all characters in the corpus onto the background image according to the drawing parameters to obtain a text image; and an instrument image enhancement module, which is used to perform different types and degrees of image enhancement on the text image using a multimodal large model to generate multiple instrument images.

[0015] According to one aspect of the present invention, an electronic device includes at least one processor and at least one memory, wherein the memory stores computer-readable instructions; the computer-readable instructions are executed by one or more of the processors, so that the electronic device implements the instrument image generation method as described above.

[0016] According to one aspect of the present invention, a storage medium stores computer-readable instructions thereon, and the computer-readable instructions are executed by one or more processors to implement the instrument image generation method as described above.

[0017] The beneficial effects brought by the technical solution provided by the present invention are:

[0018] In the above technical scheme, the present invention first obtains a real image of the instrument, identifies the font information in the real image, analyzes the text display characteristics of the instrument in different modes, builds a corpus according to the font information and the text display characteristics, uses a text detection algorithm to generate a background image according to the real image, designs character parameters according to the corpus, determines the text size according to the character parameters, determines the corpus and drawing parameters according to the text size and the corpus, draws all the characters in the corpus onto the background image according to the drawing parameters to obtain a text image, uses a multimodal large model to perform different types and degrees of image enhancement on the text image to generate multiple instrument images, and accurately identifies the real instrument Based on the font information and text display characteristics in the image, a rich corpus is constructed to ensure the accuracy of the generated image. The background image is generated by the text detection algorithm, and the character parameters are flexibly designed to achieve the perfect fusion of text and background. Image enhancement is performed through a multimodal large model, which not only enriches the image types, but also improves the image quality, enhances the image diversity and realism, and effectively improves the automation and intelligence level of instrument image generation, meeting the complex and changeable generation needs, thereby effectively solving the problems of poor diversity and uncontrollable effects of images generated by existing technologies, which are quite different from real scenes and cannot be effectively applied to real scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in describing the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without creative work.

[0020] Figure 1 is a flow chart of a method for generating an instrument image according to an exemplary embodiment;

[0021] Figure 2 It is a flowchart of a method for generating an instrument image in an application scenario;

[0022] Figure 3 is a block diagram of a device for generating an instrument image according to an exemplary embodiment;

[0023] Figure 4 is a hardware structure diagram of an electronic device according to an exemplary embodiment;

[0024] Figure 5 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0025] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0026] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present disclosure refers to the presence of the features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0027] The present invention provides an instrument image generation method, which performs image enhancement through a multimodal large model, not only enriches the image types, but also improves the image quality, enhances the diversity and realism of the image, and solves the problem that the diversity of the image generated by the prior art is poor and the effect is uncontrollable, and the image is very different from the real scene, and cannot be effectively applied to the real scene. The instrument image generation method is suitable for an instrument image generation device, which can be an electronic device. The instrument image generation method in the embodiment of the present invention can be applied to a variety of scenarios, such as instrument image generation, etc.

[0028] See also Figure 1 , an embodiment of the present invention provides a method for generating an instrument image, which is applicable to an electronic device.

[0029] In the following method embodiments, for ease of description, the execution subject of each step of the method is taken as an electronic device as an example for illustration, but this does not constitute a specific limitation.

[0030] like Figure 1 As shown, the method may include the following steps:

[0031] Step 110, obtaining a real image of the instrument, identifying font information in the real image, analyzing text display characteristics of the instrument in different modes, and constructing a corpus based on the font information and text display characteristics.

[0032] In one possible implementation, the text contents of the instrument in different modes are obtained, and statistical analysis is performed on all the text contents to obtain the character set and character frequency. The text combination rules of the instrument in different modes are sorted out according to the character set and character frequency to obtain the character combination rule. The first corpus and the second corpus are randomly generated according to the character frequency. The third corpus that conforms to the various modes of the instrument is generated according to the character combination rule. The character spacing of the first corpus, the second corpus and the third corpus are controlled and merged to obtain the corpus.

[0033] Step 130, using a text detection algorithm to generate a background image according to the real image, designing character parameters according to the corpus, and determining the text size according to the character parameters.

[0034] In one possible implementation, a text detection algorithm is used to detect text areas in real images, a vertex is randomly generated outside the text area and the width and height values ​​are randomly set, and a square centered on the vertex, a rectangle with the vertex as the upper left point, and a rectangle with the vertex as the lower right point are obtained according to the width and height values. It is calculated whether the rectangles and the text area have intersections, and the rectangles without intersections are used as background images. The background images are spliced ​​and fused to obtain the background image.

[0035] In one possible implementation, the aspect ratios of all non-intersecting rectangles are calculated and sorted according to the aspect ratios, two background images whose aspect ratio difference is less than 0.1 are randomly selected, the heights of the two background images are adjusted to be the same while maintaining their respective aspect ratios, sub-regions of equal width are selected from the right end of one background image and the left end of the other background image, weights are set for the sub-regions, and a weighted fusion algorithm is used to merge the sub-regions according to the weights to obtain a background image.

[0036] In one possible implementation, fonts and corpora are selected based on a corpus, a background image is determined and its width and height are obtained, the font size is initialized, and the ratio between the width and height of the corpus and the width and height of the background image is calculated. If the ratio is lower than 2 / 3, the font size is increased; if the ratio is higher than 2 / 3, the font size is reduced to obtain the text size.

[0037] Step 150, determining the corpus and drawing parameters according to the text size and the corpus, and drawing all characters in the corpus onto the background image according to the drawing parameters to obtain a text image.

[0038] In one possible implementation, the number and size of the corpus are obtained, the remaining interval is calculated in combination with the size of the background image, the remaining interval is divided by the integer part of the number of characters in the corpus as a fixed interval and a random interval is generated, a font thickness is generated for each character in the corpus, an image processing algorithm is used to obtain the width of the character according to the text size, the drawing position of the character is calculated according to the fixed interval, random interval, font thickness and width of the character, a grayscale value is randomly generated according to the average grayscale value of the character, and all characters in the corpus are drawn onto the background image according to the text size, font thickness and grayscale value to obtain a text image.

[0039] Among them, the random interval fluctuates around the value of the fixed interval.

[0040] Step 170: Use the multimodal large model to perform different types and degrees of image enhancement on the text image to generate multiple instrument images.

[0041] In one possible implementation, a multimodal large model is selected and initialized, a text image is selected and prompt words are defined, the text image and the corresponding prompt words are input into the multimodal large model to generate an image description, the text image is enhanced, each enhanced text image and the corresponding prompt words are input into the multimodal large model again to generate an enhanced image description, the text similarity between the image description and the enhanced image description is calculated, and if the text similarity meets the set conditions, the enhanced text image is used as an instrument image.

[0042] Among them, the enhancement processing includes lighting adjustment, adding shadows, simulating occlusion, etc., which are not limited here.

[0043] Through the above process, the present invention first obtains a real image of the instrument, identifies the font information in the real image, analyzes the text display characteristics of the instrument in different modes, builds a corpus according to the font information and the text display characteristics, uses a text detection algorithm to generate a background image according to the real image, designs character parameters according to the corpus, determines the text size according to the character parameters, determines the corpus and drawing parameters according to the text size and the corpus, draws all the characters in the corpus onto the background image according to the drawing parameters to obtain a text image, uses a multimodal large model to perform different types and degrees of image enhancement on the text image to generate multiple instrument images, and accurately identifies the real instrument Based on the font information and text display characteristics in the image, a rich corpus is constructed to ensure the accuracy of the generated image. The background image is generated by the text detection algorithm, and the character parameters are flexibly designed to achieve the perfect fusion of text and background. Image enhancement is performed through a multimodal large model, which not only enriches the image types, but also improves the image quality, enhances the image diversity and realism, and effectively improves the automation and intelligence level of instrument image generation, meeting the complex and changeable generation needs, thereby effectively solving the problems of poor diversity and uncontrollable effects of images generated by existing technologies, which are quite different from real scenes and cannot be effectively applied to real scenes.

[0044] In one application scenario, Figure 2 A flowchart of an instrument image generation method is shown.

[0045] like Figure 2 As shown, firstly, the text font recognition in the image is performed. This step is to extract text information from the image. Next, the image acquisition of the complete cycle of the instrument is carried out to ensure that complete and continuous image data is obtained. The extracted text is analyzed for rules and character sets to understand the characteristics of the text and the range of the character set. Custom background generation and corpus generation are carried out in parallel. Among them, custom background generation is to create a suitable background image according to needs, and corpus generation is to organize and analyze the extracted text to form a corpus for subsequent use. After the custom background is generated, select the most suitable background from multiple possible backgrounds as the background for subsequent processing, select the appropriate font according to the needs, select the text to be processed from the corpus, prepare for text drawing, and configure the parameters of the selected text, including font size, color, spacing, etc., to ensure that the text meets the requirements when drawing.

[0046] Sizing and positioning are the preparatory work for text drawing. Specifically, first determine the size of the text drawing, including width, height, etc., and then position it to determine the specific position of the text on the background image. After all parameters and positions are determined, draw the text onto the background image. This step is the core of the entire processing flow, combining text with images and enhancing the image with drawn text to improve image quality and visual effects.

[0047] Through the above process, the embodiment of the present invention significantly improves the extraction and display effects of text information in images through refined text font recognition and image processing processes, ensures the integrity and accuracy of text data, and enhances the pertinence and practicality of image content by in-depth analysis of text characteristics and character sets and customized generation of background and corpus. In addition, flexible configuration of text parameters such as font, size, color, etc. not only meets diverse visual needs, but also improves the readability and aesthetics of the text. Finally, through image enhancement processing, the image quality is further optimized, the text and background are perfectly integrated, the overall visual effect and information communication efficiency are enhanced, and strong support is provided for image recognition, information display and other fields.

[0048] In another application scenario, a more diverse and realistic text image is generated for a specific scenario through a custom corpus construction module, a background generation module, a font size adjustment module, a text drawing module, and an image enhancement module. Specifically, the following steps are included:

[0049] Step S1, font recognition.

[0050] Specifically, the font corresponding to the instrument display text is identified through a third-party online website (such as whatfontis), so as to provide an accurate font style for subsequent text drawing.

[0051] In the above process, the fonts used are identified from the actual instrument display image. By uploading the image to online services such as whatfontis, their algorithms are used to identify the fonts in the image and obtain the font name or similar font information, which is crucial for subsequent simulation of the same style of text display.

[0052] Step S2: corpus construction.

[0053] Specifically, based on the corpus construction module, the instrument display cycle is traversed, the text content displayed in each mode during the sorting cycle is recorded, and statistical analysis is performed on the text content to obtain the character set, character frequency and text combination rules. According to the character frequency and basic character combination rules, a rich and diverse corpus is generated for subsequent text image generation.

[0054] In the above process, the text display characteristics of the instrument in different working cycles or modes are analyzed, such as the displayed text content, format, color, etc. Then, a corpus containing various text styles and contents is constructed based on these characteristics for use in subsequent generation of simulation texts.

[0055] Step S3, generating a customized background.

[0056] Specifically, based on the background generation module, a text detection model is used to detect text areas from real images, and blank areas are automatically captured in real images to avoid overlapping with existing text. A blank background image is generated, and a large number of blank background images of different sizes, positions and combinations are generated through splicing, gradient fusion and other methods to increase background diversity.

[0057] The text detection model is a model commonly used in OCR to detect the position of text in an image, and is not limited here.

[0058] Step S4, font size adjustment.

[0059] Specifically, based on the font size adjustment module, the font size is automatically adjusted according to the corpus length and the background image size to ensure that the text can reasonably fill the area to be recognized. The image processing library (such as Pillow) is used to calculate the aspect ratio of the font at different sizes, and the optimal font size is obtained through iterative adjustment.

[0060] Step S5: text drawing.

[0061] Specifically, after selecting the font, background image and corpus, the appropriate character spacing and position are calculated based on the text drawing module, and the character spacing module and font depth module are designed to increase data diversity. The image processing library is used to draw the corpus onto the background image to simulate the text display effect of the real instrument image.

[0062] Step S6: automatic image enhancement.

[0063] Specifically, in view of the image differences caused by the instrument installation method and environment, the generated text image is enhanced by various environmental simulations such as lighting, shadow, and occlusion. Based on the automatic image enhancement module, a multimodal large model (such as CLIP) is used to control the image enhancement quality to ensure that the enhanced image is consistent with the actual scene and not overly distorted. By comparing the image descriptions before and after enhancement, effective enhanced images are automatically screened to avoid introducing noise data.

[0064] Specifically, the corpus construction module is used to automatically generate a corpus for the instrument display content, aiming to simulate or predict the text content that the instrument may display in different display cycles and modes. The following steps may be included:

[0065] Step S201, traverse the instrument display cycle, record and organize the text content displayed in each mode within the cycle, and fully understand the instrument's workflow and display cycle, including all text content that will be displayed in different time periods or operating modes. Collect the specific text content in each display cycle and mode by actually observing, recording or accessing the instrument's data interface.

[0066] Step S202, statistically analyze all text contents to obtain character sets and character frequencies, analyze the collected text data, identify all characters that appear and the frequencies of these characters in the text, use text analysis tools or write scripts to perform character segmentation and frequency statistics on the text content, and form character sets and character frequency tables.

[0067] Step S203, sort out the rules of text combination under each mode, abstract a number of basic rules of character combination, identify the patterns or rules in the text content, such as specific vocabulary combinations, formats or grammatical rules. Through manual analysis or machine learning technology, extract common character combination patterns from text data, such as date format, time display rules, specific status codes, etc.

[0068] Step S204, randomly generate corpus 1 with corresponding probability according to the character frequency; randomly generate corpus 2 with the same probability, and generate random text data based on the character frequency to simulate the text diversity that the instrument may display. Use a random number generator to randomly select characters to form text according to the character frequency table to generate two independent corpora. The same probability means that the probability of each character being selected is the same or the characters are randomly selected according to the actual frequency.

[0069] Step S205, based on the abstract basic character combination rules, generate corpus 3 that conforms to various patterns within the meter cycle, generate text data that follows specific rules and patterns, so as to more accurately simulate the actual display content of the meter. Use the character combination rules to build an algorithm or template to generate text that conforms to a specific pattern to form corpus 3.

[0070] Step S206, for corpus 1, 2 and part of corpus 3, insert a number of space characters to control the character spacing, simulating the formatting and layout in actual text display, such as spaces or tabs between characters. When generating or processing a corpus, insert spaces or tabs as needed to adjust the text format and readability.

[0071] Step S207, merging corpora 1, 2, and 3 can generate the final corpus of the meter, merging multiple corpora to form a comprehensive and diverse corpus to fully cover various text contents that may be displayed on the meter. Merging corpora 1, 2, and 3 into a single corpus also requires deduplication, sorting, or further formatting to ensure the quality and practicality of the final corpus.

[0072] In the above process, through various methods such as data analysis, rule extraction, random generation and rule constraints, a corpus generation system that can simulate the instrument display content is constructed, which provides strong support for instrument simulation, testing or data analysis.

[0073] Specifically, the background generation module is used to automatically capture blank areas from real images and generate a variety of blank background images as the basis for subsequent text drawing. Due to differences in the position and angle of the camera during installation, the instrument image in the real scene will be affected by the complex environment. In order to generate a text image close to the real instrument environment, it is necessary to automatically capture the blank area that does not contain existing text from the real image as the background. The following steps may be included:

[0074] Step S301, text area detection: Use a text detection algorithm to detect all text areas in the real image and mark these areas as S1, where S1 contains a series of rectangular boxes boxn, each boxn is defined by the coordinates of two points (xn1, yn1, xn2, yn2) from the upper left to the lower right.

[0075] Step S302, randomly generate candidate background points: randomly generate an upper left point P (x, y), which must be guaranteed not to be in any detected text area S1 to ensure its validity as a background point.

[0076] Step S303, generate candidate background rectangular frames: take point P as the center, randomly set width w and height h, and generate three rectangular frames: a square with P as the center, a rectangle with P as the upper left point, and a rectangle with P as the lower right point. These rectangular frames are used as candidate background areas.

[0077] Step S304, filter out rectangles without intersections: calculate the intersection-over-union (IoU) of the three rectangles and all text areas S1 respectively, and if any rectangle has an intersection with a text area, discard the rectangle. This step ensures that the selected rectangle is a real blank area.

[0078] Step S305, repeatedly generating diversified background images: Repeating steps S302 to S304, through multiple random generation and screening, a large number of blank background images of different sizes and positions are obtained. These background images are used as alternatives for subsequent text drawing.

[0079] Step S306, sorting and splicing: Calculate the aspect ratio of all valid rectangular frames and sort them according to the aspect ratio. Randomly select two blank background images B1 and B2 whose aspect ratio difference is less than 0.1, and adjust their heights to the same (h = (h1 + h2) / 2, while maintaining their respective aspect ratios). Select sub-areas of equal width from the right end of B1 and the left end of B2, and merge the two background images in a gradient manner based on the weighted fusion method of OpenCV to generate a more complex spliced ​​background image. Repeat the above splicing steps to obtain a large number of spliced ​​background images with different combinations, further increasing the diversity and complexity of the background image.

[0080] The purpose of sorting is to select blank background images with similar aspect ratios for splicing, so as to ensure that the spliced ​​image is more natural and harmonious visually.

[0081] In the above process, blank areas are automatically captured from real images, and a large number of diverse blank background images are obtained through the processes of generation, screening, sorting and splicing. These background images serve as the basis for subsequent text drawing, which can significantly improve the diversity and authenticity of generated text images, thereby better meeting the training needs of instrument text recognition.

[0082] Specifically, the font size adjustment module is used to automatically adjust the font size according to a given background image and corpus. When drawing text onto a background image, since the size of the background image and the length of the text corpus are different, the font size needs to be automatically adjusted to ensure that the text can be properly filled into the background image. The following steps may be included:

[0083] Step S401, select font, corpus and background image: First, determine the font to be used, the specific text corpus and the background image as a basis.

[0084] Step S402, obtaining the width and height of the background image: using image processing technology (such as Pillow library) to obtain the width (img_w) and height (img_h) of the background image.

[0085] Step S403, initializing the font size: selecting a lower initial font size (font_min) as a starting point to ensure that there is enough room to increase the font size in subsequent adjustments.

[0086] Step S404, calculating the initial text width and height: using the Pillow library or other image processing libraries, calculating the width (text_w) and height (text_h) of the corpus text under the current font and the initial font size.

[0087] Step S405, ratio judgment and adjustment: Calculate the ratio: Calculate the ratio of the text width and height to the background image width and height. This ratio can reflect the space occupied by the text in the background image under the current font size. Ratio adjustment: If the text width-to-height ratio is lower than 2 / 3 of the background image width and height, it means that the text is too small at the current font size and is not enough to fully fill the background image, so the font size needs to be increased. If the text width-to-height ratio is higher than 3 / 4 of the background image width and height, it means that the text is too large at the current font size and may exceed the range of the background image, so the font size needs to be reduced. Adjust the font size according to the judgment result, and recalculate the text width, height and ratio.

[0088] The above process is repeated until the ratio of the text width to the background image width meets a preset condition (eg, between 2 / 3 and 3 / 4). The font size (font_s) obtained at this time is the font size that meets the requirements.

[0089] In the above process, the font size is automatically adjusted so that the text can be filled into the background image without being too small or too large, thereby improving the quality and readability of the generated text image.

[0090] Specifically, the text drawing module is used to draw the text onto the background image according to the font and its size after selecting the font, background image, and text, and calculating the appropriate font size, in order to increase data diversity to meet more needs. The following steps may be included:

[0091] Step 501, obtain basic parameters: obtain the corpus number num and size (text_h, text_w), combine the size of the background image (img_h, img_w), and calculate the remaining margin. This margin is used to leave space between texts and between texts and image edges.

[0092] Step 502, calculate character spacing: Use the character spacing module to first calculate the fixed spacing m_i, which is usually set to margin divided by the integer part of the number of characters num in the corpus, that is, m_i = margin / / num, and generate a random spacing array [m_1, m_2, ..., m_num], which fluctuates around the fixed spacing to increase the diversity of text layout.

[0093] Step 503, character drawing preparation: traverse each character c_i in the corpus (from the 1st character to the numth character), and randomly generate a font thickness parameter t_i for each character to simulate text effects of different thicknesses.

[0094] Step 504, calculate the character position: use Pillow library (or other image processing library) to obtain the actual width c_wi of character c_i in the specified font and size, and calculate the drawing position c_offset_i of character c_i according to the corresponding value m_i in the fixed interval or random interval array. This position is calculated based on the position c_offset_{i-1} of the previous character, the current interval m_i, the character width c_wi and the font weight t_i, that is, c_offset_i = c_offset_{i-1} + m_i + c_wi + t_i.

[0095] For the first character, c_offset_0 is usually set to a starting position on the background image.

[0096] Step 505, setting font depth: calculate the average grayscale value of the start and end areas of the expected drawing position of the character c_i on the background image, use this as the depth lower limit, and randomly generate a grayscale value d_i, which is above the depth lower limit, to simulate the color depth changes of the text in the image and increase the realism of the image.

[0097] Step 506, drawing characters: at the calculated position c_offset_i, using the specified font size font_s, font thickness t_i and font depth (grayscale value) d_i, draw the character c_i onto the background image, and repeat the drawing process until all characters in the corpus are drawn onto the background image.

[0098] In the above process, by introducing the randomness of character spacing and the change of font depth, the diversity and realism of the generated text images are increased, thereby improving the quality of the dataset used to train the OCR model.

[0099] Specifically, the image enhancement module is used to address the problem of large differences in text images caused by instrument installation methods and environments, and limited diversity of directly generated images, to ensure that the enhanced images are diverse and meet the needs of actual scenarios, while avoiding the introduction of noise data. The following steps may be included:

[0100] Step S601, initializing a multimodal large model: selecting and initializing a suitable multimodal large model, such as CLIP (Contrastive Language-Image Pre-training), which can understand and compare the association between images and texts.

[0101] Step S602, set formatting prompt: define a formatting prompt to describe the text position and content in the image to the CLIP model. For example, prompt = "aimage with text at [c_offset_0, c_offset_num], text is [label]", where [c_offset_0, c_offset_num] represents the position coordinates of the text in the image, and [label] is the specific content of the text in the image.

[0102] Step S603, selecting a generated text image: selecting one of the generated text images for enhancement processing.

[0103] Step S604, input image and prompt word to CLIP: input the selected text image (image) and the corresponding prompt word (prompt, where [label] is replaced by the actual text content in the image) into the CLIP model. The CLIP model generates an image description (desc) based on the image and prompt word, which reflects the content and location information of the text in the image.

[0104] Step S605, image enhancement processing: Use the image enhancement module to perform various enhancement processing on the text image, such as lighting adjustment, adding shadows, simulating occlusion, etc., to enhance the diversity and authenticity of the image. The enhancement processing can be performed to different degrees to simulate different environmental conditions. Each enhancement method will generate one or more enhanced images (image_i).

[0105] Step S606, verify the validity of the enhanced image: input each enhanced image (image_i) and the original prompt word (prompt, where [label] remains unchanged) into the CLIP model again. The CLIP model generates a description of the enhanced image (desc_i). Compare the text content part (i.e., the [text] part) in the original image description (desc) and the enhanced image description (desc_i). If the two are the same or the similarity is high enough, it means that the enhancement process has not changed the core information of the text in the image, and the enhanced image is valid; otherwise, the enhanced image may introduce noise or change the text content and should be discarded.

[0106] Step S607, screening effective enhanced images: Based on the above verification results, screen out all effective enhanced images, which not only maintain the core information of the original text, but also increase diversity and authenticity, and are suitable for subsequent model training or testing.

[0107] In the above process, the quality of image enhancement can be effectively controlled to ensure that the generated text images are both diverse and close to the actual scene, providing high-quality data support for the training of the instrument text recognition model.

[0108] Through the above process, the embodiment of the present invention forms a complete text image generation process from font recognition, corpus construction, custom background generation, font size adjustment to text drawing and automatic image enhancement. Through the synergy of these steps, text images with strong diversity and close to real instrument images can be generated, providing strong support for the training of OCR recognition models. Through the organic combination of multiple modules, the problems of image generation in the prior art such as data limitation, poor diversity and uncontrollable effect are solved, providing a new and more effective data generation method for the training of text recognition models.

[0109] The following is an embodiment of the device of the present invention, which can be used to execute the instrument image generation method involved in the present invention. For details not disclosed in the embodiment of the device of the present invention, please refer to the method embodiment of the instrument image generation method involved in the present invention.

[0110] See also Figure 3 , an instrument image generating device 800 is provided in an embodiment of the present invention.

[0111] The instrument image generating device 800 includes but is not limited to: a corpus building module 810 , a data preparation module 830 , a text image generating module 850 and an instrument image enhancing module 870 .

[0112] The corpus construction module 810 is used to obtain a real image of the instrument, identify the font information in the real image, analyze the text display characteristics of the instrument in different modes, and construct a corpus based on the font information and text display characteristics.

[0113] The data preparation module 830 is used to generate a background image according to a real image using a text detection algorithm, design character parameters according to a corpus, and determine a text size according to the character parameters.

[0114] The text image generation module 850 is used to determine the corpus and drawing parameters according to the text size and the corpus, and draw all characters in the corpus onto the background image according to the drawing parameters to obtain the text image.

[0115] The instrument image enhancement module 870 is used to generate multiple instrument images by performing different types and degrees of image enhancement on the text image using a multimodal large model.

[0116] It should be noted that, when the instrument image provided in the above embodiment is generated, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the instrument image generating device will be divided into different functional modules to complete all or part of the functions described above.

[0117] In addition, the instrument image generation device and the instrument image generation method provided in the above embodiments belong to the same concept, and the specific manner in which each module performs operations has been described in detail in the method embodiments and will not be repeated here.

[0118] Figure 4 The present invention schematically illustrates the structure of an electronic device according to an exemplary embodiment.

[0119] It should be noted that the electronic device is only an example adapted to the present invention and cannot be considered to provide any limitation on the scope of use of the present invention. The electronic device cannot be interpreted as requiring dependence on or having Figure 4 One or more components of exemplary electronic device 2000 are shown.

[0120] The hardware structure of the electronic device 2000 may vary greatly due to differences in configuration or performance, such as Figure 4 As shown, the electronic device 2000 includes: a power supply 210 , an interface 230 , at least one memory 250 , and at least one central processing unit (CPU) 270 .

[0121] Specifically, the power supply 210 is used to provide operating voltage for each hardware device on the electronic device 2000 .

[0122] The interface 230 includes at least one wired or wireless network interface 231 for interacting with external devices. Of course, in other examples adapted by the present invention, the interface 230 may further include at least one serial-to-parallel conversion interface 233, at least one input-output interface 235, and at least one USB interface 237, etc. Figure 4 As shown, this is not intended to be a specific limitation.

[0123] The memory 250 is a carrier for storing resources, which may be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon include an operating system 251, an application 253 and data 255, etc. The storage method may be temporary storage or permanent storage.

[0124] Among them, the operating system 251 is used to manage and control various hardware devices and application programs 253 on the electronic device 2000 to enable the central processor 270 to calculate and process the massive data 255 in the memory 250. It can be WindowsServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0125] The application program 253 is a computer-readable instruction that performs at least one specific task based on the operating system 251, and may include at least one module ( Figure 4 (not shown), each module may respectively include computer-readable instructions for the electronic device 2000. For example, the instrument image generating device may be regarded as an application 253 deployed on the electronic device 2000.

[0126] The data 255 may be signal information, etc., stored in the memory 250 .

[0127] The central processor 270 may include one or more processors and is configured to communicate with the memory 250 through at least one communication bus to read the computer-readable instructions stored in the memory 250, thereby realizing the operation and processing of the mass data 255 in the memory 250. For example, the instrument image generation method is completed by the central processor 270 reading a series of computer-readable instructions stored in the memory 250.

[0128] In addition, the present invention can also be implemented by hardware circuits or hardware circuits combined with software. Therefore, the implementation of the present invention is not limited to any specific hardware circuits, software, or a combination of the two.

[0129] See also Figure 5 In an embodiment of the present invention, an electronic device 4000 is provided. The electronic device 4000 may include: a desktop computer, a laptop computer, a server, etc. with sensor recognition capability.

[0130] exist Figure 5 In the embodiment, the electronic device 4000 includes at least one processor 4001 and at least one memory 4003.

[0131] The data interaction between the processor 4001 and the memory 4003 can be realized through at least one communication bus 4002. The communication bus 4002 may include a path for transmitting data between the processor 4001 and the memory 4003. The communication bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The communication bus 4002 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0132] Optionally, the electronic device 4000 may further include a transceiver 4004, which may be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception, etc. It should be noted that in actual applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present invention.

[0133] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of the present invention. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0134] The memory 4003 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program instructions or codes in the form of instructions or data structures and can be accessed by the electronic device 4000, but is not limited to this.

[0135] Computer-readable instructions are stored in the memory 4003 , and the processor 4001 can read the computer-readable instructions stored in the memory 4003 through the communication bus 4002 .

[0136] The computer-readable instructions are executed by one or more processors 4001 to implement the instrument image generation method in the above-mentioned embodiments.

[0137] In addition, an embodiment of the present invention provides a storage medium on which computer-readable instructions are stored. The computer-readable instructions are executed by one or more processors to implement the instrument image generation method as described above.

[0138] A computer program product is provided in an embodiment of the present invention. The computer program product includes computer-readable instructions. The computer-readable instructions are stored in a storage medium. One or more processors of an electronic device read the computer-readable instructions from the storage medium, load and execute the computer-readable instructions, so that the electronic device implements the instrument image generation method as described above.

[0139] Compared with the related art, the present invention has the following beneficial effects:

[0140] 1. The present invention obtains a real image of the instrument, identifies the font information in the real image, analyzes the text display characteristics of the instrument in different modes, builds a corpus according to the font information and the text display characteristics, uses a text detection algorithm to generate a background image according to the real image, designs character parameters according to the corpus, determines the text size according to the character parameters, determines the corpus and drawing parameters according to the text size and the corpus, draws all the characters in the corpus onto the background image according to the drawing parameters to obtain a text image, uses a multimodal large model to perform different types and degrees of image enhancement on the text image to generate multiple instrument images, and accurately identifies the text in the real instrument image. Based on the font information and text display characteristics, a rich corpus was constructed to ensure the accuracy of the generated images, the text detection algorithm was used to generate the background image, and the character parameters were flexibly designed to achieve the perfect fusion of text and background. The multimodal large model was used for image enhancement, which not only enriched the image types, but also improved the image quality, enhanced the image diversity and realism, and effectively improved the automation and intelligence level of instrument image generation, meeting the complex and changeable generation needs, thereby effectively solving the problems that the diversity of images generated by the existing technology is poor and the effect is uncontrollable, which is quite different from the real scene and cannot be effectively applied to the real scene.

[0141] 2. The present invention can improve the generalization ability of the OCR model: by simulating and generating a large number of instrument text images close to real scenes, this scheme provides diversified data support for OCR model training, thereby significantly improving the recognition accuracy and generalization ability of the OCR model under different environments and conditions.

[0142] 3. The present invention can reduce the cost of data annotation: Traditional OCR model training requires a large amount of manually annotated data, while this solution greatly reduces the workload of manual annotation and reduces the cost of data acquisition and processing by automatically generating text images.

[0143] 4. The present invention can adapt to complex and changeable instrument environments: The characteristics of instrument text recognition are that the text fonts are standardized but changeable, and the environment in which the instrument is located is complex. This solution can generate text images that are closer to the real instrument environment through the designed custom background generation module, automatic text following module, etc., so that the trained OCR model can better adapt to actual application scenarios.

[0144] 5. The present invention can enhance data diversity: Through the automatic image enhancement module, the present solution can simulate various environmental factors such as lighting, shadow, occlusion, etc., further enhancing the diversity of generated text images and helping to improve the OCR model's recognition ability for complex scenes.

[0145] 6. The present invention can improve the efficiency of model training: Since the generated data is closer to the real scene and more diverse, the OCR model trained using this data can converge faster, thereby improving the efficiency of model training.

[0146] 7. The present invention can reduce noise data: by evaluating the image enhancement effect with a large multimodal model, the present solution can automatically screen out effective enhanced images, reduce the noise data introduced by improper enhancement, and ensure the quality of training data.

[0147] 8. The present invention is flexible and extensible: each module in the present solution, such as corpus construction, custom background generation, font size adjustment, etc., can be flexibly adjusted and optimized according to actual needs, with strong extensibility, and can adapt to different instrument types and recognition requirements. The present invention simulates and generates instrument text images close to real scenes, providing efficient and low-cost data support for OCR model training, significantly improving the generalization ability and recognition accuracy of the OCR model, and has important practical application value.

[0148] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0149] The above descriptions are only some embodiments of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for generating an instrument image, characterized in that: The method comprises: Acquire a real image of an instrument, identify font information in the real image, analyze text display characteristics of the instrument in different modes, and construct a corpus based on the font information and text display characteristics; Generate a background image based on the real image using a text detection algorithm, design character parameters based on the corpus, and determine a text size based on the character parameters; Determine the corpus and drawing parameters according to the text size and the corpus, and draw all characters in the corpus onto the background image according to the drawing parameters to obtain a text image; A multimodal large model is used to perform different types and degrees of image enhancement on the text image to generate multiple instrument images.

2. The instrument image generation method according to claim 1, characterized in that: The analyzing instrument's text display characteristics in different modes, and constructing a corpus according to the font information and the text display characteristics, include: Acquire text contents in different modes of the instrument, perform statistical analysis on all the text contents, and obtain character sets and character frequencies; Combing the text combination rules of the instrument in different modes according to the character set and the character frequency to obtain the character combination rules, and randomly generating the first corpus and the second corpus according to the character frequency; A third corpus conforming to each mode of the instrument is generated according to the character combination rule, and the first corpus, the second corpus and the third corpus are subjected to character spacing control and fusion to obtain a corpus.

3. The instrument image generation method according to claim 1, characterized in that: The step of using a text detection algorithm to generate a background image according to the real image comprises: Using a text detection algorithm to detect a text area in the real image, randomly generating a vertex outside the text area and randomly setting width and height values; According to the width and height values, a square centered on the vertex, a rectangle with the vertex as the upper left point, and a rectangle with the vertex as the lower right point are obtained respectively; It is calculated whether the rectangles and the text area have intersections, and the rectangles without intersections are used as background images. The background images are spliced ​​and fused to obtain a background image.

4. The instrument image generation method according to claim 3, characterized in that: The step of splicing and fusing the background images to obtain the background image comprises: Calculate the aspect ratios of all non-intersecting rectangles and sort them by aspect ratio, and randomly select two background images whose aspect ratio difference is less than 0.1; The heights of the two background images are adjusted to be the same while maintaining their respective aspect ratios, and sub-regions of equal width are selected from the right end of one background image and the left end of the other background image respectively; Weights are set for the sub-regions, and a weighted fusion algorithm is used to merge the sub-regions according to the weights to obtain a background image.

5. The instrument image generation method according to claim 1, characterized in that: The step of designing character parameters according to the corpus and determining the text size according to the character parameters comprises: Select fonts and corpus according to the corpus, determine the background image and obtain its width and height, initialize the font size, and calculate the ratio between the width and height of the corpus and the width and height of the background image; If the ratio is lower than 2 / 3, the font size is increased, and if the ratio is higher than 2 / 3, the font size is reduced to obtain the text size.

6. The instrument image generation method according to claim 1, characterized in that: The step of determining the corpus and drawing parameters according to the text size and the corpus, and drawing all characters in the corpus onto the background image according to the drawing parameters to obtain a text image includes: The number and size of the corpus are obtained, and the remaining interval is calculated in combination with the size of the background image. The remaining interval is divided by the integer part of the number of characters in the corpus as a fixed interval and a random interval is generated; the random interval fluctuates around the value of the fixed interval; Generating a font weight for each character in the corpus, obtaining a width of the character according to the text size using an image processing algorithm, and calculating a drawing position of the character according to a fixed interval, a random interval, a font weight, and a width of the character; A grayscale value is randomly generated according to the average grayscale value of the characters, and all the characters in the corpus are drawn onto the background image according to the text size, font thickness and grayscale value to obtain a text image.

7. The instrument image generation method according to claim 1, characterized in that: The method of using the multimodal large model to perform different types and degrees of image enhancement on the text image to generate multiple instrument images includes: Select and initialize a multimodal large model, select a text image and define prompt words, and input the text image and the corresponding prompt words into the multimodal large model to generate an image description; Performing enhancement processing on the text image, and inputting each enhanced text image and the corresponding prompt word into the multimodal large model again to generate an enhanced image description; the enhancement processing includes adjusting lighting, adding shadows, and simulating occlusion; The text similarity between the image description and the enhanced image description is calculated, and if the text similarity meets a set condition, the enhanced text image is used as an instrument image.

8. An instrument image generating device, characterized in that: The device comprises: A corpus building module, used to obtain a real image of the instrument, identify font information in the real image, analyze text display characteristics of the instrument in different modes, and build a corpus according to the font information and text display characteristics; A data preparation module, configured to generate a background image according to the real image using a text detection algorithm, design character parameters according to the corpus, and determine a text size according to the character parameters; A text image generation module, used for determining a corpus and drawing parameters according to the text size and the corpus, and drawing all characters in the corpus onto a background image according to the drawing parameters to obtain a text image; The instrument image enhancement module is used to use a multimodal large model to perform different types and degrees of image enhancement on the text image to generate multiple instrument images.

9. An electronic device, characterized in that: include: at least one processor and at least one memory, wherein: The memory has computer-readable instructions stored thereon; The computer-readable instructions are executed by one or more of the processors so that the electronic device implements the instrument image generating method according to any one of claims 1 to 7.

10. A storage medium having computer-readable instructions stored thereon, characterized in that: The computer-readable instructions are executed by one or more processors to implement the instrument image generating method according to any one of claims 1 to 7.

Images