Text generation method, electronic equipment, storage medium and program product

Abstract

The embodiment of the invention provides a text generation method, electronic equipment, a storage medium and a program product, and the method comprises the steps: determining a first image feature of a to-be-processed chart image, the first image feature comprising a character feature of a target character in the to-be-processed chart image; determining a second image feature of the to-be-processed chart image, wherein the second image feature comprises a graph feature of a target graph in the to-be-processed chart image; and generating a target text based on the first image feature and the second image feature, wherein the target text is used for describing a corresponding relationship between the target graph and the target character. Through the method and the device, the text in the chart can be accurately identified.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing, and specifically, to a text generation method and electronic device, storage medium, and program product. Background Art

[0002] At present, with the development of artificial intelligence, there is a demand for question-and-answering of document information, and documents will include charts. When users ask questions about the information in the chart, the chart needs to be recognized as structured text. However, the recognition of charts in related technologies cannot describe the relationship between charts and text, resulting in the inability to accurately recognize the content in the chart. Summary of the invention

[0003] The present invention provides a text generation method, an electronic device, a storage medium, and a program product, so as to at least solve the technical problem in the related art that the text in the chart cannot be accurately recognized.

[0004] According to one aspect of the present invention, there is provided a text generation method, comprising: determining a first image feature of a chart image to be processed, the first image feature comprising text features of a target text in the chart image to be processed; determining a second image feature of the chart image to be processed, the second image feature comprising graphic features of a target graphic in the chart image to be processed; generating a target text based on the first image feature and the second image feature, the target text being used to describe the correspondence between the target graphic and the target text.

[0005] According to another aspect of the present invention, a text generation device is provided, comprising: a first determination module, used to determine a first image feature of a chart image to be processed, wherein the first image feature includes a text feature of a target text in the chart image to be processed; a second determination module, used to determine a second image feature of the chart image to be processed, wherein the second image feature includes a graphic feature of a target graphic in the chart image to be processed; and a first generation module, used to generate a target text based on the first image feature and the second image feature, wherein the target text is used to describe the correspondence between the target graphic and the target text.

[0006] According to another embodiment of the present application, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0007] According to another embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any one of the above method embodiments when running.

[0008] According to another embodiment of the present application, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0009] Through this application, by extracting the text features of the target text and the graphic features of the target graphic in the chart image to be processed, the content in the chart image to be processed can be more comprehensively understood, thereby improving the accuracy of recognition, converting the chart content into a text format that is easy to operate and apply, and providing convenience for further data analysis and application. Therefore, the technical problem of the inability to accurately recognize the text in the chart in the related technology can be solved, and the effect of accurately recognizing the text in the chart can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 It is a hardware structure block diagram of a mobile terminal of a text generation method according to an embodiment of the present application;

[0011] Figure 2 is a flowchart of a text generation method according to an embodiment of the present application;

[0012] Figure 3 is a schematic diagram of a line chart image according to an embodiment of the present application;

[0013] Figure 4 is a flow chart for identifying a line chart image according to a specific embodiment of the present application;

[0014] Figure 5 It is a structural block diagram of a text generation device according to an embodiment of the present application. DETAILED DESCRIPTION

[0015] The embodiments of the present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0016] It should be noted that the terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0017] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 1 is a hardware structure block diagram of a mobile terminal of a text generation method according to an embodiment of the present application. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0018] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the text generation method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the mobile terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0019] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0020] In this embodiment, a text generation method is provided. Figure 2 is a flowchart of a text generation method according to an embodiment of the present application, such as Figure 2 As shown, the process includes the following steps:

[0021] Step S202: determining a first image feature of the chart image to be processed, wherein the first image feature includes a text feature of a target text in the chart image to be processed.

[0022] Optionally, the chart image to be processed is a visual image used to display and analyze data, which can make the data more clearly understood and interpreted. The chart image to be processed includes target text and target graphics, and the types of target graphics include but are not limited to: bar charts, column charts, line charts, pie charts, scatter charts, etc. For example, Figure 3 The image of the chart to be processed including a line chart is shown.

[0023] Optionally, before determining the first image feature of the chart image to be processed, this embodiment needs to first obtain the chart image to be processed. The method of obtaining the chart image to be processed is not limited. For example, the chart image to be processed can be obtained by: using chart software to create a chart and exporting it as an image file; using an online chart tool to create and export a chart image; using a screenshot tool to capture a screenshot of the chart; downloading the chart from a website.

[0024] Optionally, the first image feature is mainly used to describe the text features of the target text in the chart image to be processed. The text features of the target text include but are not limited to: the coordinates of the target text in the chart image to be processed, the size of the target text (including height, width, thickness), font style, font size, color, language, style, texture, connectivity (including the connection relationship between the strokes of the target text). These features can accurately identify and analyze the text content in the chart image to be processed, so as to better understand the information expressed by the chart image to be processed.

[0025] Step S204, determining a second image feature of the chart image to be processed, where the second image feature includes a graphic feature of a target graphic in the chart image to be processed.

[0026] Optionally, the second image feature is mainly used to describe the visual features of the target graphic in the chart image to be processed. The visual features refer to the features used to identify and locate the chart image to be processed. These features can help the computer vision model accurately identify and locate the target graphic in the chart image to be processed. The visual features include but are not limited to the color features, texture features, shape features, edge features, depth features, optical flow features, etc. of the target graphic. These features can be used alone or in combination to improve the accuracy of identifying the chart image to be processed.

[0027] Optionally, the above-mentioned visual features can be obtained through a network model (for example, a convolutional neural network model, a neural network model based on an attention mechanism). The acquisition steps include but are not limited to: 1) Preprocessing the chart image to be processed, for example, scaling, rotating or normalizing to eliminate the influence of factors such as noise and illumination changes. 2) Feature extraction: Extract color features, texture features, shape features, edge features, depth features, optical flow features, etc. from the preprocessed image. 3) Feature dimensionality reduction: Since the number of extracted features may be large, in order to improve computational efficiency and avoid overfitting, the features need to be reduced in dimensionality. 4) Feature fusion: Combine the reduced features into a unified feature vector.

[0028] Optionally, the second image feature can also be obtained in the following manner: 1) subjecting the chart image to be processed to Gaussian blur with a probability of 0.5 and Gaussian noise with a probability of 0.5; 2) using the additive color model RGB to randomly switch channels of the processed image, adding a random value between 0 and 10 to each pixel with a probability of 0.3, or subtracting a random value between 0 and 10 from each pixel with a probability of 0.3, to obtain an enhanced chart image; 3) subjecting the enhanced chart image to a Patch encoding module to obtain an image code.

[0029] Among them, after the enhanced chart image passes through the Patch encoding module, the enhanced chart image will be converted into a digital code. This digital code can be used to represent the characteristics and structure of the enhanced chart image, so that the enhanced chart image can be processed and analyzed by a computer program. The Patch encoding module includes pix2struct and vit, which can convert each part of the enhanced chart image into a digital vector, thereby realizing the encoding and representation of the image.

[0030] Step S206: Generate target text based on the first image feature and the second image feature, where the target text is used to describe the corresponding relationship between the target graphic and the target text.

[0031] Optionally, the target text in this embodiment can be part of the graphic component of the chart image to be processed, or it can be an independent text description. The correspondence between it and the target graphic includes the following aspects: content correspondence, the target text directly describes or explains the content of the target graphic. For example, if the target graphic is a bar graph, the target text may be a legend or an explanation of the data in the graph. Functional correspondence, the target text guides the user on how to interact with the graphic, such as a button label or prompt information. Visual correspondence, the target text and the target graphic may correspond to each other through visual elements such as color, font, position, etc. to enhance the overall aesthetics and consistency. Logical correspondence, the target text is a further explanation or expansion of the target graphic data, providing a logical connection to help users better understand the meaning behind the graphic.

[0032] Optionally, the target text is a readable text description, through which the corresponding relationship between the target text and the target graphic in the chart image to be processed can be reflected, that is, the target text can reflect the meaning to be expressed by the target graphic, and can further refine and accurately express the content in the target text. The specific expression of the target text is not limited.

[0033] For example, according to Figure 3 The chart image to be processed shown can obtain the target text 1 in the following text form:

[0034] Title | Spot Price: Average Price: Wheat:

[0035] 17-05|2614;

[0036] 17-11|2546;

[0037] 18-05|2399;

[0038] 18-11|2490;

[0039] 19-05|2423;

[0040] 19-11|2400;

[0041] 20-05|2440;

[0042] 20-11|2488;

[0043] 21-05|2540.

[0044] Among them, the title part in the above text 1 corresponds to Figure 3 Legend for (-Spot Price: Average Price: Wheat), the numbers correspond to Figure 3 The values ​​of the horizontal and vertical axes corresponding to the line chart in .

[0045] For example, according to Figure 3 The chart image to be processed shown can obtain the target text 2 in the form of Table 1:

[0046] Table 1:

[0047]

[0048]

[0049] The above target text 1 and target text 2 have the same content, both of which are used to describe Figure 3The extracted content includes the title Spot Price: Average Price: Wheat, in which the horizontal axis values ​​include: 17-05, 17-11, 18-05, 18-11, 19-05, 19-11, 20-05, 20-11, 21-05; the vertical axis values ​​include: 2614, 2546, 2399, 2490, 2423, 2400, 2440, 2488, 2540.

[0050] Specifically, taking 17-05|2614 as an example, 17-05 means the time May 2017, 2614 means in May 2017, the spot price: average price: wheat is 2614, and so on.

[0051] For another example, the target text can also be structured data, that is, data stored in a fixed format that can be organized, stored, and analyzed in a predefined way. The characteristic of structured data is that it has a fixed data model and can be easily sorted, searched, and processed.

[0052] The execution subject of the above steps may be a terminal, a server, a specific processor provided in the terminal or the server, or a processor or processing device provided relatively independently from the terminal or the server, but is not limited thereto.

[0053] Through the above steps, by extracting the text features of the target text and the graphic features of the target graphic in the chart image to be processed, the content in the chart image to be processed can be more comprehensively understood, thereby improving the accuracy of recognition, converting the chart content into a text format that is easy to operate and apply, and providing convenience for further data analysis and application. Therefore, the technical problem of the inability to accurately recognize the text in the chart in the related technology can be solved, and the effect of accurately recognizing the text in the chart can be achieved.

[0054] In an exemplary embodiment, determining a first image feature of a chart image to be processed comprises the following steps:

[0055] Step S302, extracting N text blocks from the chart image to be processed, wherein the N text blocks include the target text, and N is a positive integer;

[0056] Optionally, the value of N can be flexibly set based on the actual application scenario or the requirements for text block extraction. For example, the value can be 2, 4, 6, 8, 10..., without any limitation here. In an embodiment of the present application, the target text included in the chart image to be processed can be divided into 8 text blocks.

[0057] Optionally, this embodiment can extract N text blocks from the chart image to be processed through the following steps: S1, pre-processing the chart image to be processed: adjusting the size of the chart image to be processed, for example, adjusting the chart image to be processed to a suitable resolution. De-noising the chart image to be processed, for example, using a filter (for example, Gaussian blur, median filter, etc.) to remove noise in the chart image to be processed. S2, detecting the text area in the chart image to be processed: a text detection algorithm can be used to detect the area including text in the chart image to be processed. S3, extracting the text area: extracting the corresponding text block from the chart image to be processed according to the detected text area. S4, performing text segmentation on the text block, dividing it into text blocks including single characters or words. S5, adjusting the order of the text blocks: the text blocks can be sorted by analyzing the relative position and direction between the text blocks, or extracting the order of the text blocks. S6, identifying and processing N text blocks: selecting the first N text blocks from the sorted text blocks. S7, identifying the text in the text block: optical character recognition technology can be used to perform text recognition on the selected text block. S8, result output: output the recognized text block into a required format (eg, text file, JSON format, etc.).

[0058] Step S304, performing a text encoding operation on the N text blocks to obtain text encoding features, and determining the text type of the target text according to the text encoding features;

[0059] Optionally, this embodiment can perform text encoding operations on the N above-mentioned text blocks through the following steps to obtain text encoding features, and determine the text type of the above-mentioned target text according to the above-mentioned text encoding features: identify the above-mentioned target text contained in the N above-mentioned text blocks, and splice the target text in the N above-mentioned text blocks in a preset order to obtain a spliced ​​text; convert the above-mentioned spliced ​​text into characters in a second preset format to obtain spliced ​​characters; perform vector conversion on the above-mentioned spliced ​​characters to obtain a feature vector; encode the above-mentioned feature vector to obtain the above-mentioned text encoding features of the above-mentioned target text; determine the text type of the above-mentioned target text based on the above-mentioned text encoding features.

[0060] Optionally, this embodiment can determine the text type of the target text based on the above-mentioned text encoding features. For example, the correspondence between the text encoding features and the text type includes but is not limited to: ASCII encoding features: uppercase and lowercase English letters: AZ, az; numbers: 0-9, special characters: such as spaces, punctuation marks, etc.; text type: ASCII character set, applicable to English text. GB2312 encoding features: simplified Chinese characters; text type: simplified Chinese character set, applicable to simplified Chinese text. GBK encoding features: simplified Chinese characters, including characters in GB2312; extended simplified Chinese character set; text type: GBK character set, applicable to simplified Chinese text, including some uncommon characters. GB18030 encoding features: simplified Chinese characters, including characters in GB2312 and GBK; extended simplified Chinese character set, including part of traditional Chinese. Text type: GB18030 character set, suitable for simplified Chinese text, including some extended characters; UTF-8 encoding features: compatible with ASCII-encoded English characters; can represent characters in multiple languages, such as Chinese, Japanese, Korean, etc.; Text type: Unicode character set, suitable for text in multiple languages. Through the above correspondence, the text type of the target text can be determined according to different text encoding features. For example, if a text uses ASCII encoding, it is likely to be English text.

[0061] Optionally, this embodiment can identify the target text contained in the N text blocks and splice the target text in the N text blocks in a preset order to obtain a spliced ​​text by the following steps: splice the N text blocks in the preset order to obtain a spliced ​​text block; identify the target text in the spliced ​​text block to obtain the spliced ​​text; or, identify the N text blocks and determine the target text in the N text blocks; splice the target text in the N text blocks in the preset order to obtain the spliced ​​text. The preset order includes the original arrangement order of the N text blocks in the chart image to be processed, or the arrangement order of the target text in the chart image to be processed.

[0062] Optionally, in the chart image, the original arrangement order of the N above-mentioned text blocks is determined according to their positions in the image. For example, the title is usually located at the top of the chart, the axis labels are at the bottom or both sides of the chart, and the legend is on one side of the chart. The arrangement order of the target text in the chart image to be processed refers to the position of the target text in the chart, and they are arranged in a certain logical order, such as time order, numerical order, etc. The original arrangement order is based on the specific position of the text block in the image, while the arrangement order of the target text is based on the logical order of the information or data reflected by the text content. The original arrangement order serves more for the layout and aesthetics of the chart, while the arrangement order of the target text is to more effectively convey information and data relationships. The original arrangement order may remain consistent in different chart designs (for example, the title is always placed at the top), while the arrangement order of the target text may vary depending on the chart type and the data displayed. When splicing N text blocks, the accurate target text can be obtained by combining the arrangement order of the N text blocks.

[0063] In this embodiment, optical character recognition (OCR) technology can be used to recognize the target text contained in the N text blocks. The preset splicing order can be the order of extracting the N text blocks. For example, the text blocks are extracted from the chart image to be processed in a left-to-right order. When splicing the text in the N text blocks, the splicing is also performed in a left-to-right order. Thus, a correctly ordered text combination can be obtained. In addition, after the text combination is obtained by splicing, the text combination can also be post-processed, including: removing spaces, spelling checking, context correction, formatting, etc. A language model can also be used for further optimization. When determining the splicing characters in this embodiment, whether the method of first combining N text blocks and then recognizing the text is adopted, or the method of first recognizing the text and then combining is adopted, an accurate text combination can be obtained. The splicing characters can be either text or numbers.

[0064] This embodiment can quickly extract useful information from a large amount of data by automatically combining scattered text blocks in a specific order. The combined text is more likely to form a coherent narrative or description, making it easier to read and understand.

[0065] Step S306, performing a coordinate encoding operation on the coordinates of the N text blocks in the chart image to be processed to obtain coordinate features of the target text;

[0066] Optionally, this embodiment can perform a coordinate encoding operation on the coordinates of the N text blocks in the chart image to be processed to obtain the coordinate features of the target text through the following steps: for each text block, calculate the coordinates of multiple target position points in the text block according to the coordinates of the text block in the chart image to be processed, wherein the multiple target position points include the border position of the text block; encode the coordinates of the multiple target position points to obtain multiple coordinate codes; merge the multiple coordinate codes to obtain a coordinate code sequence; determine the coordinate features of the text block according to the coordinate relationship between the coordinate codes in the coordinate code sequence; determine the coordinate features of the target text according to the coordinate features of the N text blocks.

[0067] Optionally, in this embodiment, to calculate the coordinates of multiple preset position points in a text block, it is necessary to first determine the coordinates of the text block, and then calculate its coordinates based on the relative positions of the preset position points. For example, if the coordinates of the text block are (x, y), and the positions of the preset position points relative to the text block are (dx1, dy1), (dx2, dy2), ..., then the coordinates of the preset position points can be calculated using the following formula: the coordinates of preset position point 1 are (x+dx1, y+dy1), and the coordinates of preset position point 2 are (x+dx2, y+dy2);

[0068] For example, if the coordinates of the text block are (100, 200), and the positions of the preset points relative to the text block are (20, 30) and (-10, 50), then we can calculate:

[0069] The coordinates of the preset position point 1 are (100+20, 200+30)=(120, 230), and the coordinates of the preset position point 2 are (100-10, 200+50)=(90, 250).

[0070] Through this embodiment, the coordinates of the preset position point can be calculated according to the coordinates of the text block and the relative position of the preset position point.

[0071] Optionally, in this embodiment, the coordinates of the plurality of target position points are encoded to obtain a plurality of coordinate codes in order to convert the numerical information of the coordinate points into a format or standard for further processing.

[0072] Coordinate point encoding methods include, but are not limited to, decimal representation, hexadecimal representation, binary representation, Base64 encoding, URL encoding, custom format, etc. The selection of encoding method depends on the specific application scenario and requirements. For example, in this embodiment, binary or hexadecimal representation is used for processing the chart image to be processed.

[0073] Optionally, in this embodiment, multiple coordinate codes are merged to obtain a coordinate code sequence, which can be achieved by merging multiple coordinate codes into the coordinate features of a text block. This can be achieved by calculating the average, maximum, minimum, standard deviation and other statistics of the coordinate codes. For example, if there are multiple coordinate codes (x1, y1), (x2, y2), ..., (xn, yn), their average (μx, μy) and standard deviation (σx, σy) can be calculated, and then these statistics can be combined into a feature vector (μx, μy, σx, σy). This embodiment combines multiple coordinate codes to accurately extract the overall position features of the text block.

[0074] Step S308, determining the sizes of the N text blocks, and performing size encoding operations on the sizes of the N text blocks to obtain size features of the target text;

[0075] Optionally, this embodiment can determine the sizes of the N text blocks mentioned above through the following steps, and perform size encoding operations on the sizes of the N text blocks mentioned above to obtain size characteristics of the target text: based on the coordinates of the N text blocks mentioned above in the above-mentioned chart image to be processed, calculate the sizes of the N text blocks mentioned above; normalize the sizes of the N text blocks mentioned above to convert the sizes of the N text blocks mentioned above into N target sizes within a preset range; encode the N target sizes above to obtain size characteristics of the text blocks mentioned above.

[0076] Optionally, for example, to calculate the size of a text block in a chart image to be processed, the coordinate information of the text block in the chart image to be processed can be obtained, including the coordinate values ​​of the upper left corner and the lower right corner. Use the coordinate values ​​of the upper left corner and the lower right corner of the text block to calculate the width and height of the text block. The calculation formula is as follows: width = the horizontal coordinate value of the lower right corner - the horizontal coordinate value of the upper left corner; height = the vertical coordinate value of the lower right corner - the vertical coordinate value of the upper left corner. Based on the calculated width and height, the size information of the text block can be obtained. For example, if the coordinates of the upper left corner of the text block are (100, 200) and the coordinates of the lower right corner are (300, 400), then the width of the text block can be calculated to be 300-100=200, and the height is 400-200=200, so the size information of the text block is 200x200.

[0077] Optionally, in this embodiment, the size encoding operation is to convert the size information into a specific encoding format for easy processing and analysis. When the size features of the text block are obtained, the size encoding operation includes the following steps: S1, extracting the size information from the text block, such as width, height, etc. S2, converting the extracted size information into a specific encoding format, which can be a numerical encoding or a categorical encoding, for subsequent processing and analysis. S3, based on the encoded size information, the size features of the text block can be extracted, such as the average size, maximum size, minimum size, etc. S4, based on the extracted size features, further analysis can be performed, such as size distribution, size change trend, etc. Through the size encoding operation and extraction of size features, the size characteristics of the text block can be better understood, providing more information and basis for subsequent processing and analysis.

[0078] Step S310: determining the text type, coordinate features and size features of the target text as the first image features.

[0079] Optionally, since the text type, coordinate features and the above-mentioned size features of the target text are characters after encoding processing, they can be merged to obtain the first image feature. The merging method can be to merge the N coordinate features and the N size features into an N*4 matrix. Then, the type feature and the N*4 matrix are merged together to obtain the first image feature. In this way, all features can be integrated together to facilitate subsequent processing and analysis.

[0080] This embodiment extracts N text blocks from the chart image to be processed, and performs text encoding operations, coordinate encoding operations, and size encoding operations on the N text blocks to obtain accurate first image features, thereby improving the accuracy of generating the target text.

[0081] In an exemplary embodiment, the generating of the target text based on the first image feature and the second image feature includes:

[0082] Step S402, combining the first image feature and the second image feature to obtain a target image feature;

[0083] Optionally, this embodiment can combine the above-mentioned first image feature and the above-mentioned second image feature to obtain the target image feature through the following steps: map the above-mentioned first image feature and the above-mentioned second image feature to a preset feature space to obtain the first feature variable of the above-mentioned first image feature and the second feature variable of the above-mentioned second image feature; splice the above-mentioned first feature variable and the above-mentioned second feature variable to obtain the above-mentioned target image feature; or, splice the above-mentioned first image feature and the above-mentioned second image feature to determine the spliced ​​image feature; map the above-mentioned spliced ​​image feature to the preset feature space to obtain the above-mentioned target image feature.

[0084] Optionally, the first image feature and the second image feature are mapped to a preset feature space, and the dimension of the preset feature space and the feature mapping method need to be determined. If the dimension of the preset feature space is n, a linear transformation can be used to map the first image feature and the second image feature to the preset feature space. If the first image feature is x1 and the second image feature is x2, the two features can be mapped to the preset feature space using a linear transformation matrix A, that is: y1=Ax1, y2=Ax2, where y1 and y2 are the mapping results of the first image feature and the second image feature in the preset feature space, respectively. In this way, the first image feature and the second image feature can be converted into new feature variables, thereby facilitating subsequent processing and analysis. Mapping the first image feature and the second image feature to the preset feature space also requires preprocessing the first image feature and the second image feature, for example, standardization or normalization, and then mapping them to the preset feature space using the selected feature transformation method.

[0085] Optionally, the first feature variable and the second feature variable are spliced ​​to obtain the target image feature by combining the two feature variables in a certain order or rule to obtain a new feature variable. This new feature variable can be used to represent the feature of the target image, so as to be used for subsequent analysis and processing. The splicing method can be a simple connection of two feature variables, or a new feature variable obtained by certain operations or processing. The spliced ​​target image feature can contain more information, which helps to describe and analyze the target image more comprehensively and accurately. The first image feature and the second image feature are spliced ​​to obtain the spliced ​​image feature. The splicing method is the same as that of the above embodiment, which will not be repeated here. The above spliced ​​image feature is mapped to a preset feature space to obtain the above target image feature. The mapping method and the processing of the feature variables are the same as those of the above embodiment, which will not be repeated here.

[0086] This embodiment maps the first image feature and the second image feature to a preset feature space for vector splicing, or splices first and then maps, to obtain the target image feature, which can achieve the purpose of quickly and accurately performing variable transformation on the image feature.

[0087] Step S404: decode the target image features to obtain the target code.

[0088] Optionally, the purpose of decoding the target image feature is to obtain the target text encoding that is the same as the text encoding in the target vocabulary to facilitate subsequent search. The decoding method needs to be selected according to the format of the target image feature.

[0089] This embodiment can quickly search for characters matching the target text encoding from the target vocabulary by decoding the target image features.

[0090] Step S406, searching the target vocabulary for characters matching the target code to obtain the target text.

[0091] Optionally, before the present embodiment searches for text matching the target encoding from the target vocabulary to obtain the target text, the method further includes: identifying the target text included in the chart image to be processed; performing word segmentation processing on the target text to obtain M word segments, where M is a positive integer; storing the word segmentation encodings of the M word segments into a preset vocabulary to obtain the target vocabulary, where the preset vocabulary includes multiple pre-stored text encodings.

[0092] Optionally, optical character recognition (OCR) technology can be used to identify the target text included in the chart image to be processed. For example, the text in the chart image to be processed can be converted into an editable text format using OCR software or tools to identify the target text. The text in the chart image to be processed can also be identified and extracted by using computer vision and text recognition algorithms. The text in the chart image to be processed can also be pattern matched and identified using a string matching algorithm to obtain target text information. For example, the chart image to be processed is input into the OCR interface of the OCR tool to obtain all the text on the chart image to be processed. All the words are passed through the word segmentation interface to obtain all the word segments.

[0093] For example, if the text I love programming is obtained, it will be divided into three words: I, love, programming, and three word segmentation codes ocab0. Phrases allow the model to learn text semantic information. For example, define all Chinese characters supported by the OCR interface as vocab1 (each character is defined as a word to ensure that all Chinese characters can be recognized). Add vocab0+vocab1 to the preset vocabulary to expand the preset vocabulary.

[0094] This embodiment can further expand the target word list by segmenting the text in the chart image to be processed, so that the target text can be found quickly and accurately.

[0095] The present application is described below in conjunction with specific embodiments:

[0096] This specific embodiment is described by taking the recognition of a chart image containing Chinese as an example. Figure 3 The line graph shown in the figure is processed to convert the line graph into structured data, and then processed by downstream tasks. Figure 4 As shown, it is a processing flow chart of this specific embodiment, including the following:

[0097] S41, collecting Chinese chart images to obtain a line chart;

[0098] S42, input the line graph into the OCR interface to obtain all the text and coordinates in the line graph.

[0099] S43, all the text blocks are passed through the word segmentation interface to obtain all the word segmentation vocab0 (for example, I love programming, which will be divided into three words: I, love, programming. Phrases allow the model to learn the semantic information of the text). All Chinese characters supported by the OCR interface are defined as vocab1 (each character is defined as a word to ensure that all Chinese characters can be recognized). Add vocab0+vocab1 to the vocabulary of the model to expand the vocabulary.

[0100] S44, splicing the text blocks obtained by the line graph through the OCR interface, converting the text in the text blocks into numbers, and obtaining the text feature embedding_char (corresponding to the text type mentioned above) through text encoding.

[0101] S45, from the coordinates of the text block on the line graph, calculate the four-point coordinates of the text block and the coordinates of the center of the text block box, and obtain the coordinate features embedding_left, embedding_right, embedding_top, embedding_bottom, and embedding_center through the encoding module respectively.

[0102] S46, using the coordinates of the text block on the line graph, calculate the width and height of the text block, and perform position encoding respectively to obtain width and height features embedding_width and embedding_height.

[0103] S47, add all the above features to obtain embedding_text. embedding_text=embedding_char+embedding_left+embedding_right+embedding_top+embedding_bottom+embedding_center+embedding_width+embedding_height.

[0104] S48, the line graph is subjected to Gaussian blur with a probability of 0.5 and Gaussian noise with a probability of 0.5, RGB channels are switched randomly, and each pixel is added with a random value between 0 and 10 with a probability of 0.3, or each pixel is subtracted with a random value between 0 and 10 with a probability of 0.3. The enhanced image is obtained. The enhanced image is subjected to Patch encoding (the method of converting images to numbers in pix2struct and vit) to obtain the image encoding embedding_visual.

[0105] S49, concatenate the image encoding embedding_visual feature and embedding_text feature to obtain the final target image feature.

[0106] S410, encoding the target image features, and then decoding the target image features to obtain text encoding features.

[0107] S411, the number of categories of the fully connected layer of text classification is modified to the number of the vocabulary after the vocabulary is added (for example, if the original supported 20,000 words and 50,000 words are added, the modified number is 70,000 categories), and the model is iterated until convergence. During reasoning, the category id is passed through the Chinese tokenizer (converting numbers into text), and the Chinese reasoning content of the line chart is obtained, as shown in Table 1.

[0108] This embodiment can recognize Chinese characters in a line graph by passing the line graph into the RCR interface. The information of the text block (e.g., top, bottom, left, right, center point, width, height) and semantic information can be used to obtain more image feature information. Thus, the association between the graph and the characters in the graph image can be accurately recognized.

[0109] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0110] In the present embodiment, a text generation device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware is also possible and conceived.

[0111] Figure 5 is a structural block diagram of a text generation device according to an embodiment of the present application, such as Figure 5 As shown, the device comprises:

[0112] A first determining module 52, used to determine a first image feature of the chart image to be processed, wherein the first image feature includes a text feature of a target text in the chart image to be processed;

[0113] A second determining module 54 is used to determine a second image feature of the chart image to be processed, wherein the second image feature includes a graphic feature of a target graphic in the chart image to be processed;

[0114] The first generating module 56 is used to generate a target text based on the first image feature and the second image feature, wherein the target text is used to describe the corresponding relationship between the target graphic and the target text.

[0115] Through the above device, by extracting the text features of the target text and the graphic features of the target graphic in the chart image to be processed, the content in the chart image to be processed can be understood more comprehensively, thereby improving the accuracy of recognition, converting the chart content into a text format that is easy to operate and apply, and providing convenience for further data analysis and application. Therefore, the technical problem of the inability to accurately recognize the text in the chart in the related technology can be solved, and the effect of accurately recognizing the text in the chart can be achieved.

[0116] In an exemplary embodiment, the first determination module 52 includes: a first extraction unit, used to extract N text blocks from the chart image to be processed, wherein the N text blocks include the target text, and N is a positive integer; a first encoding unit, used to perform a text encoding operation on the N text blocks to obtain text encoding features, and determine the text type of the target text according to the text encoding features; a second encoding unit, used to perform a coordinate encoding operation on the coordinates of the N text blocks in the chart image to be processed to obtain the coordinate features of the target text; a third encoding unit, used to determine the sizes of the N text blocks, and perform a size encoding operation on the sizes of the N text blocks to obtain the size features of the target text; the first determination unit, used to determine the text type, coordinate features and size features of the target text as the first image features.

[0117] In an exemplary embodiment, the above-mentioned first encoding unit includes: a first recognition submodule, used to identify the target text contained in the N text blocks, and splice the target text in the N text blocks in a preset order to obtain a spliced ​​text; a first conversion submodule, used to convert the spliced ​​text into characters in a second preset format to obtain spliced ​​characters; a second conversion submodule, used to perform vector conversion on the spliced ​​characters to obtain a feature vector; a first encoding submodule, used to encode the feature vector to obtain the text encoding feature of the target text; a first determination submodule, used to determine the text type of the target text based on the text encoding feature.

[0118] In an exemplary embodiment, the above-mentioned first recognition submodule includes: a first splicing subunit, used to splice the N text blocks in accordance with the preset order to obtain a spliced ​​text block; a first recognition subunit, used to identify the target text in the spliced ​​text block to obtain the spliced ​​text; or, a second recognition subunit, used to identify the N text blocks and determine the target text in the N text blocks; the first splicing subunit splices the target text in the N text blocks in accordance with the preset order to obtain the spliced ​​text.

[0119] In an exemplary embodiment, the above-mentioned second encoding unit includes: a first calculation submodule, used to calculate the coordinates of multiple target position points in the text block for each of the text blocks according to the coordinates of the text block in the chart image to be processed, wherein the multiple target position points include the border position of the text block; a second encoding submodule, used to encode the coordinates of the multiple target position points to obtain multiple coordinate codes; a first merging submodule, used to merge the multiple coordinate codes to obtain a coordinate code sequence; a second determination submodule, used to determine the coordinate features of the text block according to the coordinate relationship between the coordinate codes in the coordinate code sequence; a third determination submodule, used to determine the coordinate features of the target text according to the coordinate features of N text blocks.

[0120] In an exemplary embodiment, the third encoding unit includes: a second calculation submodule, used to calculate the sizes of the N text blocks based on the coordinates of the N text blocks in the chart image to be processed; a first processing submodule, used to normalize the sizes of the N text blocks to convert the sizes of the N text blocks into N target sizes within a preset range; and a third encoding submodule, used to encode the N target sizes to obtain size features of the text blocks.

[0121] In an exemplary embodiment, the first generation module includes: a first merging unit, used to merge the first image feature and the second image feature to obtain a target image feature; a first decoding unit, used to decode the target image feature to obtain a target code; and a first search unit, used to search for text matching the target code from a target vocabulary to obtain the target text.

[0122] In an exemplary embodiment, the above-mentioned first merging unit includes: a first mapping submodule, used to map the first image feature and the second image feature to a preset feature space to obtain a first feature variable of the first image feature and a second feature variable of the second image feature; a first splicing submodule, used to splice the first feature variable and the second feature variable to obtain the target image feature; or, a second splicing submodule, used to splice the first image feature and the second image feature to determine the spliced ​​image feature; a second mapping submodule, used to map the spliced ​​image feature to a preset feature space to obtain the target image feature.

[0123] In an exemplary embodiment, the above-mentioned device also includes: a first recognition module, which is used to search for text matching the target code from a target vocabulary, and identify the target text included in the chart image to be processed before obtaining the target text; a first processing module, which is used to perform word segmentation processing on the target text to obtain M word segments, where M is a positive integer; a first storage module, which is used to store the word segmentation codes of the M word segments into a preset vocabulary to obtain the target vocabulary, and the preset vocabulary includes multiple pre-stored text codes.

[0124] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0125] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any one of the above method embodiments are implemented.

[0126] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0127] The embodiments of the present application also provide a computer program, which includes computer instructions, which are stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps of any one of the above method embodiments.

[0128] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0129] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0130] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0131] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0132] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0133] Obviously, those skilled in the art should understand that the above modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0134] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principles of the present application shall be included in the protection scope of the present application.

Claims

1. A text generation method, characterized in that: include: Determine a first image feature of a chart image to be processed, wherein the first image feature includes a text feature of a target text in the chart image to be processed; Determining a second image feature of the chart image to be processed, wherein the second image feature includes a graphic feature of a target graphic in the chart image to be processed; A target text is generated based on the first image feature and the second image feature, where the target text is used to describe the corresponding relationship between the target graphic and the target text.

2. The method according to claim 1, characterized in that The step of determining the first image feature of the chart image to be processed comprises: Extracting N text blocks from the to-be-processed chart image, wherein the N text blocks include the target text, and N is a positive integer; Performing a text encoding operation on the N text blocks to obtain text encoding features, and determining the text type of the target text according to the text encoding features; Performing a coordinate encoding operation on the coordinates of the N text blocks in the to-be-processed chart image to obtain coordinate features of the target text; Determine the sizes of the N text blocks, and perform size encoding operation on the sizes of the N text blocks to obtain size features of the target text; The text type, coordinate features and size features of the target text are determined as the first image features.

3. The method according to claim 2, characterized in that The performing of a text encoding operation on the N text blocks to obtain a text encoding feature, and determining the text type of the target text according to the text encoding feature, comprises: Identify the target text contained in the N text blocks, and splice the target text in the N text blocks in a preset order to obtain a spliced ​​text; Converting the concatenated text into characters in a second preset format to obtain concatenated characters; Performing vector conversion on the concatenated characters to obtain a feature vector; Encoding the feature vector to obtain the text encoding feature of the target text; The character type of the target character is determined based on the character encoding feature.

4. The method according to claim 3, characterized in that The identifying the target text contained in the N text blocks and splicing the target text in the N text blocks in a preset order to obtain a spliced ​​text includes: splicing the N text blocks in the preset order to obtain a spliced ​​text block; Recognize the target text in the concatenated text block to obtain the concatenated text; or, Identify N of the text blocks and determine the target text in the N of the text blocks; The target texts in the N text blocks are spliced ​​together in the preset order to obtain the spliced ​​text.

5. The method according to claim 2, characterized in that: The performing of coordinate encoding operation on the coordinates of the N text blocks in the to-be-processed chart image to obtain the coordinate features of the target text includes: For each of the text blocks, according to the coordinates of the text block in the chart image to be processed, the coordinates of a plurality of target position points in the text block are calculated, wherein the plurality of target position points include the border position of the text block; Encoding the coordinates of the plurality of target position points to obtain a plurality of coordinate codes; Combining a plurality of the coordinate codes to obtain a coordinate code sequence; Determining the coordinate features of the text block according to the coordinate relationship between the coordinate codes in the coordinate code sequence; The coordinate features of the target text are determined according to the coordinate features of the N text blocks.

6. The method according to claim 2, characterized in that The determining the sizes of the N text blocks and performing a size encoding operation on the sizes of the N text blocks to obtain the size features of the target text includes: Calculating the sizes of the N text blocks based on the coordinates of the N text blocks in the chart image to be processed; Normalizing the sizes of the N text blocks to convert the sizes of the N text blocks into N target sizes within a preset range; The N target sizes are encoded to obtain size features of the text block.

7. The method according to claim 1, characterized in that The generating the target text based on the first image feature and the second image feature includes: Combining the first image feature and the second image feature to obtain a target image feature; Decoding the target image features to obtain a target code; The target text is obtained by searching the target vocabulary for characters matching the target encoding.

8. The method according to claim 7, characterized in that Combining the first image feature and the second image feature to obtain a target image feature includes: Mapping the first image feature and the second image feature to a preset feature space to obtain a first feature variable of the first image feature and a second feature variable of the second image feature; splicing the first feature variable and the second feature variable to obtain the target image feature; or, splicing the first image feature and the second image feature to determine a spliced ​​image feature; The spliced ​​image features are mapped to a preset feature space to obtain the target image features.

9. The method according to claim 7, characterized in that: Before searching the target word list for characters matching the target code to obtain the target text, the method further comprises: Recognizing the target text included in the chart image to be processed; Perform word segmentation on the target text to obtain M word segments, where M is a positive integer; The word segmentation codes of the M word segments are stored in a preset word list to obtain the target word list, wherein the preset word list includes a plurality of pre-stored text codes.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method described in any one of claims 1 to 9 are implemented.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the method described in any one of claims 1 to 9 when executed by a processor.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 9 are implemented.

Images