Image processing method and device, equipment and readable medium

Abstract

The embodiment of the invention discloses an image processing method and device, equipment and a readable medium, and the method comprises the steps: carrying out the text recognition processing of a virtual resource product image, and obtaining a plurality of text segments; if it is detected that text features corresponding to multiple continuous text segments in the multiple text segments are matched with data extraction types corresponding to sub-rules in the information extraction rule sequence one by one, the multiple continuous text segments serve as the multiple text segments matched with the information extraction rule sequence; performing information extraction on each matched text segment based on each sub-rule in the information extraction rule sequence to obtain extracted product description information corresponding to each sub-rule; and merging the product description information extracted by the sub-rules corresponding to the same product attribute to obtain virtual resource product information contained in the virtual resource product image. According to the technical scheme provided by the embodiment of the invention, accurate virtual resource product information can be extracted from the image.

Description

Technical Field

[0001] This application relates to the fields of computer and communication technology, and more specifically, to image processing methods, data push devices, electronic devices, computer-readable storage media, and computer program products. Background Technology

[0002] With the booming development of internet finance, more and more individuals and companies are purchasing and managing virtual resource products through applications or platforms. To improve online management efficiency, it is often necessary to identify images containing virtual resource products to extract accurate information. Optical Character Recognition (OCR) is used to extract this information, but the extracted information is not always accurate. Therefore, how to extract accurate virtual resource product information from images is a pressing problem that needs to be solved. Summary of the Invention

[0003] Embodiments of this application provide an image processing method, an image processing apparatus, an electronic device, a computer-readable storage medium, and a computer program product that can extract accurate virtual resource product information from an image.

[0004] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part from practice of this application.

[0005] According to one aspect of the embodiments of this application, an image processing method is provided, the method comprising:

[0006] The virtual resource product image is subjected to text recognition processing to obtain multiple text segments; wherein, the multiple text segments are arranged according to their positions in the virtual resource product image;

[0007] If the text features corresponding to multiple consecutive text segments in the multiple text segments are detected to match the data extraction types corresponding to the sub-rules in the information extraction rule sequence, then the multiple consecutive text segments are regarded as multiple text segments that match the information extraction rule sequence; wherein, the product description information extracted by a sub-rule is used to describe a product attribute.

[0008] Based on each sub-rule in the information extraction rule sequence, information is extracted from each matched text segment to obtain the product description information extracted by each sub-rule.

[0009] By merging the product description information extracted from sub-rules corresponding to the same product attribute, the virtual resource product information contained in the virtual resource product image is obtained.

[0010] According to one aspect of the embodiments of this application, an image processing apparatus is provided, the apparatus comprising a text recognition unit, a rule matching unit, and an information extraction unit, wherein:

[0011] The text recognition unit is used to perform text recognition processing on virtual resource product images to obtain multiple text segments;

[0012] The rule matching unit is configured to, if it detects that the text features corresponding to multiple consecutive text segments in the multiple text segments match the data extraction types corresponding to the sub-rules in the information extraction rule sequence, then regard the multiple consecutive text segments as multiple text segments that match the information extraction rule sequence; wherein, the product description information extracted by a sub-rule is used to describe a product attribute.

[0013] The information extraction unit is used to extract information from each matched text segment based on each sub-rule in the information extraction rule sequence, and to obtain the product description information extracted by each sub-rule.

[0014] The information extraction unit is further configured to merge the product description information extracted by sub-rules corresponding to the same product attribute to obtain the virtual resource product information contained in the virtual resource product image.

[0015] According to one aspect of the embodiments of this application, an electronic device is provided, including one or more processors; and a storage device for storing one or more computer programs, which, when executed by the one or more processors, cause the electronic device to implement the image processing method as described above.

[0016] According to one aspect of the embodiments of this application, the embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor of an electronic device, causes the electronic device to perform the image processing method as described above.

[0017] According to one aspect of the embodiments of this application, this application provides a computer program product, including a computer program stored in a computer-readable storage medium, wherein a processor of an electronic device reads from the computer-readable storage medium and executes the computer program, causing the electronic device to perform the image processing method as described above.

[0018] In the technical solution provided in the embodiments of this application, by finding multiple text segments that match the information extraction rule sequence, text segments containing information unrelated to virtual resource products can be filtered out. This avoids the impact of text segments containing invalid information on the information extraction process, thus improving the accuracy of information extraction. Furthermore, considering that virtual resource product information appears continuously in an image, detecting whether multiple consecutive text segments match the sub-rules in the information extraction rule sequence effectively avoids information extraction errors caused by changes in the arrangement format of virtual resource product information in the image, leading to more accurate extraction of virtual resource product information. Finally, the embodiments of this application further merge the product description information extracted by sub-rules corresponding to the same product attribute, allowing information belonging to the same product attribute but truncated into multiple text segments during text recognition to be restored to complete information. This avoids information loss or confusion in virtual resource product information, further facilitating the extraction of more accurate virtual resource product information.

[0019] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0021] Figure 1 This is a schematic diagram of the structure of an image processing system provided in an embodiment of this application;

[0022] Figure 2 This is a schematic flowchart of an image processing method provided in an embodiment of this application;

[0023] Figure 3 This is a schematic diagram of a text segment matching process provided in an embodiment of this application;

[0024] Figure 4 This is a flowchart illustrating another image processing method provided in an embodiment of this application;

[0025] Figure 5 This is a schematic diagram of a virtual resource product information extraction strategy provided in an embodiment of this application;

[0026] Figure 6This is a schematic diagram of the management interface of a product management application provided in an embodiment of this application;

[0027] Figure 7 This is a schematic diagram of an image upload interface provided in an embodiment of this application;

[0028] Figure 8 This is a schematic diagram of a virtual resource product image provided in an embodiment of this application;

[0029] Figure 9 This is a schematic diagram of a product addition success interface provided in an embodiment of this application;

[0030] Figure 10 This is a structural block diagram of an image processing apparatus illustrated in an exemplary embodiment of this application;

[0031] Figure 11 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown. Detailed Implementation

[0032] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0033] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0034] The flowcharts shown in the accompanying diagrams are merely illustrative and do not necessarily include all content and operations, nor do they necessarily have to be executed in the described order. For example, some operations may be broken down, while others may be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0035] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0036] It should also be noted that "multiple" as mentioned in this application refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0037] With the continuous development of internet technology, artificial intelligence (AI) technology has also seen significant advancements. AI technology refers to the theories, methods, techniques, and application systems that utilize digital computers or computers-controlled machines to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science; it primarily aims to understand the essence of intelligence and produce new intelligent machines that can react in a manner similar to human intelligence, enabling these machines to possess multiple functions such as perception, reasoning, and decision-making.

[0038] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, pre-trained model technology, operating / interactive systems, and mechatronics. Among these, pre-trained models, also known as large-scale models or foundational models, can be widely applied to downstream tasks across various AI fields after fine-tuning. AI software technologies primarily include computer vision (CV), speech processing, natural language processing (NLP), and machine learning / deep learning.

[0039] Computer vision is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in tasks such as target recognition, tracking, and measurement, and further performs image processing to create images more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Large model technology has brought significant changes to the development of computer vision technology. Pre-trained models in the field of vision, such as Swin-Transformer (Hierarchical VisionTransformer using Shifted Windows, a deep learning architecture based on the Transformer model), ViT (vision transformer, a visual processing model based on the Transformer architecture), and V-MOE (VisionMoE, a sparse version of Vision Transformer), can be quickly and widely applied to specific downstream tasks after fine-tuning.

[0040] Specifically, computer vision technology typically includes image processing, image recognition, image semantic understanding, image retrieval, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0041] At the same time, with the rapid development of internet technology, internet finance has also flourished. More and more individuals and companies are starting to purchase and manage virtual resource products through applications or platforms. To improve online management efficiency and enhance the user experience of these applications or platforms, artificial intelligence technology is typically employed.

[0042] Specifically, when managing virtual resource products, applications or platforms often need to use computer vision technology to identify images containing virtual resource products in order to extract accurate information about them. Related technologies often combine open-source computer vision libraries (OpenCV) and optical character recognition (OCR) to extract virtual resource product information from images.

[0043] However, this method can only accurately identify text information from images with fixed formats or simple content. The accuracy is not high for images with variable formats and complex content, such as virtual resource product images. As a result, the virtual resource product information extracted from virtual resource product images is currently inaccurate.

[0044] Based on this, this application provides an image processing scheme. After performing text recognition processing on a virtual resource product image to obtain multiple text segments, the scheme does not directly combine the multiple recognized text segments into virtual resource product information; instead, it searches for multiple text segments that match the information extraction rule sequence from the multiple text segments; finally, it extracts information from the matched multiple text segments based on the information extraction rule sequence, thereby obtaining the virtual resource product information contained in the virtual resource product image.

[0045] Virtual resource products can be products derived from resource data that can serve as a general equivalent, or products that can be traded electronically and serve as a general equivalent; there is no limitation on this. For example, virtual resource products can be funds, bonds, stocks, etc.

[0046] The information extraction rule sequence used to extract virtual resource product information can contain multiple sub-rules. The product description information extracted by a sub-rule describes a product attribute. For example, when the virtual resource product is a fund, it can contain fund product information with multiple product attributes such as fund name, holding income, and holding yield.

[0047] Because the data types of virtual resource product information differ depending on the product attribute—for example, the data type of a fund name can be a string, while the data type of holding returns can be a floating-point number—each sub-rule will also have a corresponding data extraction type, which is the data type corresponding to the relevant product attribute.

[0048] The process of finding matching multiple text segments can include: if the text features corresponding to multiple consecutive text segments in the multiple text segments are detected and match the data extraction types corresponding to the sub-rules in the information extraction rule sequence, then the multiple consecutive text segments are regarded as multiple text segments that match the information extraction rule sequence.

[0049] Specifically, the text features of each text segment can be defined as the data features of the characters contained in each text segment. For example, if the text segment "xxx fund" contains multiple characters, then the corresponding data feature is a string; if the text segment "0.98" contains numbers, specifically numbers with decimals, then the corresponding data feature could be a floating-point number.

[0050] Furthermore, considering that information belonging to the same product attribute may be truncated into multiple text segments in a virtual resource product image, the specific process of extracting information from multiple matched text segments based on the information extraction rule sequence may include: extracting information from each matched text segment based on each sub-rule in the information extraction rule sequence to obtain the product description information extracted by each sub-rule; and merging the product description information extracted by the sub-rules corresponding to the same product attribute to obtain the virtual resource product information contained in the virtual resource product image.

[0051] It is clear that the information extraction rule sequence is used to extract information about virtual resource products, and the data extraction types of each sub-rule in the information extraction rule sequence are also used to extract virtual resource product information for each product attribute. All extracted information is valid information related to virtual resource products. Therefore, based on the identified multiple text segments, this solution first finds multiple text segments that match the information extraction rule sequence, thereby filtering out text segments containing information irrelevant to virtual resource products. This avoids the impact of text segments containing invalid information on the information extraction process, ensuring that subsequent information extraction only needs to be performed on text segments containing valid virtual resource product information, thus facilitating the extraction of more accurate virtual resource product information.

[0052] Meanwhile, considering that regardless of format changes, relevant information about virtual resource products will appear continuously in the image in order to allow the target audience to quickly understand the virtual resource product, this solution will accurately obtain multiple text segments containing virtual resource product information by detecting whether multiple consecutive text segments match the sub-rules in the information extraction rule sequence. This can effectively avoid the problem of information extraction errors caused by changes in the arrangement format of virtual resource product information in the image, and is conducive to extracting more accurate virtual resource product information.

[0053] Finally, this solution addresses the issue that information belonging to the same product attribute in virtual resource product images may be truncated into multiple text segments during text recognition. It further merges the product description information extracted by sub-rules corresponding to the same product attribute, allowing the truncated information to be restored to complete information. This effectively avoids information loss in virtual resource product information and facilitates the extraction of more accurate virtual resource product information.

[0054] Based on the above image processing scheme, this application provides an image processing system, which can be found in [reference needed]. Figure 1 , Figure 1The image processing system shown may include terminal devices 101 and servers 102. The number of terminal devices 101 and servers 102 may be multiple. A communication connection is established between any terminal device and any server. For example, terminal device 101 may include any one or more of the following: smartphone, tablet, laptop, desktop computer, smart voice interaction device, smart home appliance, vehicle terminal, aircraft, and smart wearable device. Terminal device 101 may have various clients installed, such as live streaming clients, social networking clients, shopping clients, payment clients, and map clients. Server 102 may be a server or server cluster providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Any terminal device 101 and any server 102 may communicate directly or indirectly via wired or wireless communication; this application does not impose any limitations on this.

[0055] In some embodiments, the above image processing method may be performed solely by Figure 1 The image processing system shown executes the following process from server 102: Server 102 performs text recognition processing on the virtual resource product image, obtaining multiple text segments. Then, if server 102 detects that the text features corresponding to multiple consecutive text segments match the data extraction types corresponding to the sub-rules in the information extraction rule sequence, then these consecutive text segments are considered as matching text segments in the information extraction rule sequence. The product description information extracted by one sub-rule is used to describe a product attribute. Afterwards, server 102 performs information extraction on each matched text segment based on each sub-rule in the information extraction rule sequence, obtaining the product description information extracted by each sub-rule. Finally, server 102 can merge the product description information extracted by sub-rules corresponding to the same product attribute to obtain the virtual resource product information contained in the virtual resource product image.

[0056] Optionally, the above image processing method can also be achieved solely by... Figure 1 The terminal device 101 in the image processing system shown executes the process, and its specific execution process can be found in the specific execution process of the server 102, which will not be repeated here.

[0057] In other embodiments, the image processing method described above can be run in an image processing system, which may include a terminal device and a server. Specifically, the image processing method described above can be... Figure 1The image processing system shown is jointly implemented by terminal device 101 and server 102. The specific execution process is as follows: Terminal device 101 acquires virtual resource product images and uploads them to server 102. Then, server 102 performs text recognition processing on the received virtual resource product images, obtaining multiple text segments. Subsequently, if server 102 detects that the text features corresponding to multiple consecutive text segments match the data extraction types corresponding to the sub-rules in the information extraction rule sequence, then these consecutive text segments are considered as matching text segments in the information extraction rule sequence; wherein, the product description information extracted by one sub-rule is used to describe a product attribute. Then, server 102 can extract information from each matched text segment based on each sub-rule in the information extraction rule sequence, obtaining the product description information extracted by each sub-rule. Finally, server 102 can merge the product description information extracted by sub-rules corresponding to the same product attribute to obtain the virtual resource product information contained in the virtual resource product image.

[0058] It should be noted that the embodiments of this application can be applied to various scenarios where virtual resource products exist, including but not limited to cloud technology, AI (Artificial Intelligence), smart cities, smart finance and other scenarios. They can also be used in payment applications, financial applications, wealth management applications and any other applications that need to manage virtual resource products, and there are no limitations on this.

[0059] It should be noted that, in the specific implementation of this application, if the virtual resource product images and other related data or information involve objects, when the embodiments of this application are applied to specific products or technologies, permission or consent from the objects is required, and the collection, use and processing of related data or information must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0060] The following details the various implementation details of the technical solutions in the embodiments of this application:

[0061] like Figure 2 As shown, Figure 2 This is a schematic flowchart illustrating an embodiment of an image processing method, which can be applied to... Figure 1 The image processing system shown can be executed by a terminal device or a server, or by both a terminal device and a server. In this embodiment, the method is described as being executed by a server. The image processing method may include steps S201 to S204, which are described in detail below:

[0062] S201. Perform text recognition processing on the virtual resource product image to obtain multiple text segments.

[0063] In this embodiment, the virtual resource product image can specifically be an image containing virtual resource products and their related information. Specifically, the virtual resource product image can be a screenshot of the virtual resource product interface in an application, or a photograph of the virtual resource product interface; the virtual resource product image can also be obtained in other ways, which are not limited here. It should be noted that the term "application" here refers broadly to applications running on various terminal devices.

[0064] The text recognition processing of virtual resource product images can be achieved by calling OpenCV, OCR, or machine learning models in the field of vision, etc., which are not limited here.

[0065] Furthermore, the identified text segments can be arranged according to their positions within the virtual resource product image. Specifically, regardless of whether they are arranged horizontally, vertically, or in other ways, the relevant information of the virtual resource product will always be adjacent to each other in the image. Therefore, to avoid confusion in the extraction of information from different product attributes of the same virtual resource product, the identified text segments can be arranged according to their positions within the virtual resource product image.

[0066] Optionally, considering that the display format of virtual resource product information varies in different applications, different text arrangement methods can be pre-set for different applications in order to facilitate the extraction of more accurate virtual resource product information. Then, based on the text arrangement method corresponding to the source application of the virtual resource product image and the position of each text segment in the virtual resource product image, the identified multiple text segments are arranged.

[0067] For example, if the virtual resource product interface in an application displays virtual resource product information horizontally, the text arrangement method could include first arranging the text segments from largest to smallest according to their vertical positions to obtain multiple text groups, and then arranging the text segments within each text group according to their horizontal positions. The text arrangement methods for vertical or other arrangements are similar and will not be elaborated upon here.

[0068] Optionally, each pair of identified text segments can be separated by a newline to form a text list.

[0069] S202. If the text features corresponding to multiple consecutive text segments in multiple text segments are detected and match the data extraction types corresponding to the sub-rules in the information extraction rule sequence one by one, then the multiple consecutive text segments are regarded as multiple text segments that match the information extraction rule sequence.

[0070] In this embodiment of the application, the information extraction rule sequence contains multiple sub-rules, which are arranged in a certain order; the product description information extracted by a sub-rule is used to describe a product attribute.

[0071] The information extraction rule sequence can be obtained by finding corresponding sub-rules based on information about multiple product attributes of the virtual resource products that need to be used, and then flexibly combining them. Therefore, in order to accurately extract all the virtual resource product information that needs to be used, the multiple consecutive text segments in this embodiment can be some or all of the identified multiple text segments; and the sub-rules in the information extraction rule sequence refer to all the sub-rules in the information extraction rule sequence.

[0072] In practice, the information extraction rule sequence can be set using regular expressions or similar methods. Regular expressions are a powerful text processing tool that uses strings to describe rules, matches a series of texts that conform to the syntax rules, and supports extracting subexpressions that meet the requirements, effectively improving text processing efficiency.

[0073] Optionally, since step S201 mentions that the identified multiple text segments can be arranged according to their positions in the virtual resource product image, correspondingly, multiple sub-rules in the information extraction rule sequence can also be arranged according to the product attributes to which the product description information corresponding to each position in the virtual resource product image belongs.

[0074] In one embodiment, since the display format of virtual resource product information in the virtual resource product interface of different applications is different, and the virtual resource product information required in the virtual resource product interface of different applications is different, one or more information extraction rule sequences can be set according to the format of the virtual resource product interface of different applications and the needs of different applications.

[0075] Optionally, since different applications may have multiple types of virtual resource product interfaces—such as interfaces showing virtual resource products that an object has purchased, interfaces showing virtual resource products that an object has selected but not yet purchased, etc.—one or more information extraction rule sequences can be set based on the format of different virtual resource product interfaces in different applications and the needs of different applications.

[0076] Before step S202, the application identifier of the source application of the virtual resource product image can be obtained first. Then, according to the pre-defined correspondence between the application identifier and the information extraction rule sequence, the target information extraction rule sequence corresponding to the obtained application identifier is searched, and the virtual resource product information contained in the virtual resource product image is extracted based on the target information extraction rule sequence. That is, multiple text segments that match the target information extraction rule sequence are subsequently found, and information is extracted from the multiple matching text segments based on the target information extraction rule sequence to obtain the virtual resource product information contained in the virtual resource product image.

[0077] In one possible implementation, since the aforementioned approach allows for setting corresponding information extraction rule sequences for different application interfaces and requirements, the generation process of the information extraction rule sequence can specifically include: determining multiple key product attributes in the virtual resource product interface of a specified application; wherein, the specified application is the application that needs to extract virtual resource product information through the information extraction rule sequence, and the multiple key product attributes are selected from the product attributes to which the product description information contained in the virtual resource product interface belongs, based on the information requirements of the product management application or object for the virtual resource product interface. For example, key product attributes may include product name, rate of return, holding amount, etc.

[0078] Then, the data extraction types corresponding to each key product attribute are analyzed; and the number of times the product description information corresponding to each key product attribute is repeated and the information distribution order in the virtual resource product interface are obtained; finally, based on the data extraction types corresponding to each key product attribute, sub-rules matching each key product attribute are generated; and based on the number of times the information is repeated and the information distribution order corresponding to each key product attribute, the sub-rules corresponding to each key product attribute are arranged to obtain the information extraction rule sequence corresponding to the specified application.

[0079] In this process, regular expressions used to extract information from each data extraction type can be pre-edited as sub-rules corresponding to each data extraction type; then, each data extraction type and its corresponding sub-rules are stored in the rule database; then, when generating sub-rules that match each key product attribute, it is only necessary to query the rule database by data extraction type.

[0080] Optionally, you can also use automated generation tools or models to generate sub-rules that match the data extraction types corresponding to each key product attribute. For example, when the data extraction type is floating-point numbers, since the data logic for floating-point numbers is that there is a number before and after the "." sign, the sub-rule generated based on the floating-point data logic could be "\d+\.\d+". Here, "\d" is used to match and extract a single numeric character, and "+" indicates that the preceding character (in this case, a number) can appear once or multiple times.

[0081] Therefore, this application embodiment abstracts general, reusable data logic for different data extraction types to generate different sub-rules. This allows for flexible and rapid configuration of multiple sub-rules for different applications or interfaces with different formats within those applications, resulting in information extraction rule sequences corresponding to different applications. This effectively improves the access efficiency of new applications using the information extraction rule sequences for information extraction. Furthermore, new applications can not only quickly access and use the information extraction rule sequences to extract virtual resource product information, but the information extraction rule sequences for different applications can also be added or removed in a timely manner. This method of timely expansion and modification for different applications and situations greatly improves the scalability of this information extraction solution.

[0082] Furthermore, sub-rules set for a data extraction type can be flexibly reused in different information extraction rule sequences; correspondingly, the code written for the sub-rules can also be reused, and the same method can be used to maintain the duplicate code; this greatly improves code reusability and maintainability, and also facilitates the rapid integration of new applications. This reusable and configurable capability replaces the original siloed stacking of functions, which helps improve the success rate and accuracy of information extraction.

[0083] Furthermore, before the information extraction rule sequence corresponding to a specified application is put into use, the actual virtual resource product information in the virtual resource product interface of the specified application can be obtained. Then, the virtual resource product information extracted from the image containing the virtual resource product interface based on the information extraction rule sequence corresponding to the specified application is compared with the actual virtual resource product information to obtain the comparison result. If the comparison result is used to indicate information matching, the information extraction rule sequence corresponding to the specified application is put into use. If the comparison result is used to indicate information mismatch, a rule generation failure message for the specified application is generated to prompt the regeneration of the information extraction rule sequence or to check the currently generated information extraction rule sequence.

[0084] Optionally, when the number of target information extraction rule sequences corresponding to the obtained application identifier includes multiple, multiple text segments matching each target information extraction rule sequence can be found first; then, based on each target information extraction rule sequence, information extraction is performed on the multiple matching text segments to obtain the virtual resource product information included in the virtual resource product image.

[0085] Optionally, since line breaks can also be added between every two of the multiple text segments recognized as mentioned above to form a text list; therefore, a line break extraction rule can also be inserted between every two sub-rules in the information extraction rule sequence to extract the line breaks between every two text segments in the text list.

[0086] For example, the information extraction rule sequence corresponding to the application "Small Payment" is shown in Table 1 below:

[0087]

[0088] Among them, "\n" is used to represent the extraction of line breaks, and "(" and ")" are the markers for the start and end of sub-rules respectively. The first line of the information extraction rule sequence corresponding to "Small Payment" contains five sub-rules. "[^.+\n]+" is the first sub-rule, "[]" defines a character set, "^" represents negation, and "." matches any single character (except the line break). Therefore, the sub-rule "[^.+\n]+" is used to match and extract one or more consecutive characters that are not a dot (.), a plus sign (+), or a line break.

[0089] "[+-]?\d+\.\d+" is the second sub-rule. "[+-]" is used to match and extract a plus sign or a minus sign, and "?" indicates that the previous character (here, the set of plus sign or minus sign) is optional, that is, the plus sign or minus sign can appear zero or one time. "\d" is used to match and extract a numeric character, and "+" indicates that the previous character (here, the number) can appear one or more times. "\." is used to match and extract a dot ".", and in a regular expression, a dot is a special character, so it needs to be escaped with a backslash. The "\d+" at the end is the same as the previous "\d+". Therefore, the sub-rule "[+-]?\d+\.\d+" is used to match a floating-point number that may have a plus sign or a minus sign.

[0090] The third and fourth sub-rules are the same as the second sub-rule and are also used to match and extract a floating-point number that may have a plus sign or a minus sign. The fifth sub-rule is also similar to the second sub-rule, except that the fifth sub-rule contains a "%" for matching and extracting the percent sign character. The logical rules of other information extraction sub-rules are similar to the above information extraction sub-rules and will not be elaborated here.

[0091] Specifically, when the virtual resource product is a fund, the first sub-rule in the first row of information extraction rule sequence corresponding to "small payment" can be used to extract the product attribute of the fund name, the second sub-rule can be used to extract the product attribute of the holding amount, the third sub-rule can be used to extract the product attribute of the holding profit, the fourth sub-rule can be used to extract the product attribute of yesterday's profit, and the fifth sub-rule can be used to extract the product attribute of the holding rate of return.

[0092] Optionally, as can be seen from the foregoing descriptions of text features and data extraction types, if the text features of a text segment are the same as or similar to the data extraction type corresponding to the sub-rule, then it can be determined that the text segment matches the sub-rule.

[0093] For practical applications, please refer to the appendix. Figure 3 This illustrates a text segment matching process. For example... Figure 3 As shown, after taking a screenshot of the fund product interface of the "small payment" application, a fund product image 301 can be obtained; then, after performing text recognition processing on the fund product image 301, a text list 302 containing multiple text segments is obtained.

[0094] After obtaining the four information extraction rule sequences corresponding to the "Mini Pay" application, the text segments matching each information extraction rule sequence are searched from text list 302. During the search, no matching text segments were detected for the first three information extraction rule sequences. Since the fourth information extraction rule sequence (i.e., rule 303) has the following sub-rules: the first sub-rule extracts strings that are not periods (.), plus signs (+), or newlines; the second sub-rule extracts floating-point numbers that may contain positive or negative signs; the third sub-rule extracts floating-point numbers that may contain positive or negative signs; and the fourth sub-rule extracts floating-point numbers that may contain positive or negative signs and also have a percent sign, the text segments in lines 11 to 14 and lines 15 to 18 of text list 302 match the fourth information extraction rule sequence (i.e., rule 303).

[0095] In one possible implementation, due to the different content formats in the product interfaces of different applications or different virtual resources within the same application, other fixed content prompt text may be inserted between consecutive product description information.

[0096] Therefore, to improve the matching accuracy between the information extraction rule sequence and the text segment, and thus obtain more accurate virtual resource product information, different redundant text segments can be pre-defined for different applications or different application interfaces; then, redundant text segments are deleted from the identified multiple text segments, resulting in processed multiple text segments; subsequently, if the text features corresponding to multiple consecutive text segments in the processed multiple text segments are detected and matched one-to-one with the data extraction types corresponding to the sub-rules in the information extraction rule sequence, then the consecutive multiple text segments are considered as multiple text segments that match the information extraction rule sequence.

[0097] The redundant text segment can be set based on text that may be inserted between continuous product description information in different applications or application interfaces. For example, Figure 3 The system may display text such as "Holding Amount," "Holding Profit," and "Holding Return Rate" next to the relevant figures for each fund product. The corresponding text list might look like the following:

[0098] 15:08

[0099] Small payment

[0100] Total assets (yuan)

[0101] Estimated earnings today (in RMB) 2678.71

[0103] +0.00

[0104] Product Name

[0105] Information Industry Stock A

[0106] Holding amount

[0107] Portfolio returns

[0108] Return on holdings 10.26

[0110] +0.26

[0111] +2.60%

[0112] BBB Era Stock A

[0113] Holding amount

[0114] Portfolio returns

[0115] Return on holdings 1266.72

[0117] -326.21

[0118] -20.48%

[0119] Market Analysis: A small, consecutive rise is imminent!

[0120] Therefore, redundant text segments for the "small payment" application could include "holding amount", "holding profit", and "holding return rate". Thus, the text segments "holding amount", "holding profit", and "holding return rate" in the above text list can be deleted.

[0121] S203. Based on each sub-rule in the information extraction rule sequence, extract information from each matched text segment to obtain the product description information extracted by each sub-rule.

[0122] In this embodiment of the application, since the multiple text segments identified as mentioned above can also be combined into a text list, the specific process of information extraction may include: reading the characters in each matched text segment based on each sub-rule in the information extraction rule sequence until a newline character is read; and combining the characters read before the newline character into the product description information extracted by each sub-rule.

[0123] In one embodiment, since a virtual resource product image may contain virtual resource product information from multiple virtual resource products, multiple consecutive text segments may be detected multiple times, each corresponding to a different text feature, which is then matched one-to-one with the data extraction type corresponding to a sub-rule in the information extraction rule sequence. The information extraction rule sequence is only used to extract virtual resource product information from a single virtual resource product across multiple product attributes.

[0124] To avoid confusion and misattribution of virtual resource product information among different virtual resource products, and thus improve the accuracy of virtual resource product information extraction, a method can be used. This involves identifying consecutive text segments from multiple text segments that match the information extraction rule sequence as the text sequence corresponding to each virtual resource product. Then, information extraction is performed on the text sequences corresponding to each virtual resource product based on the information extraction rule sequence, thereby obtaining the virtual resource product information for each virtual resource product.

[0125] S204. By merging the product description information extracted from sub-rules corresponding to the same product attribute, the virtual resource product information contained in the virtual resource product image is obtained.

[0126] As mentioned above, different information extraction rule sequences can be set for different application interface formats. These formats may include text line breaks, text column breaks, etc., which will cause product description information belonging to the same product attribute to be truncated into multiple text segments during text recognition. Therefore, the information extraction rule sequence set for this type of format may have multiple sub-rules for extracting product description information belonging to the same product attribute.

[0127] Therefore, in order to make the product description information of the same product attribute more complete, in this embodiment of the application, the product description information extracted by the sub-rules corresponding to the same product attribute will be merged, so that the truncated information can be restored to complete information, thereby effectively avoiding the situation of missing or chaotic information in virtual resource product information, and facilitating the extraction of more accurate virtual resource product information.

[0128] In this embodiment, by finding multiple text segments that match the information extraction rule sequence, text segments containing information unrelated to virtual resource products can be filtered out. This avoids the impact of text segments containing invalid information on the information extraction process, thus improving the accuracy of information extraction. Furthermore, considering that virtual resource product information appears continuously in an image, detecting whether multiple consecutive text segments match multiple consecutive sub-rules effectively avoids information extraction errors caused by changes in the arrangement format of virtual resource product information in the image. Compared to traditional solutions where minor adjustments to the virtual resource product interface easily lead to failure in recognizing the corresponding virtual resource product image, this embodiment is unaffected by the interface format, facilitating the extraction of more accurate virtual resource product information. Finally, this embodiment further performs precise information extraction on the matched multiple text segments according to the information extraction rule sequence, further enhancing the extraction of more accurate virtual resource product information.

[0129] Furthermore, in this embodiment, different information extraction rule sequences can be set for different applications, and the sub-rules in the information extraction rule sequences corresponding to different applications can be reused when the extraction logic is the same. This makes it possible to configure only the information extraction rule sequence when a new application needs to identify and extract virtual resource product information, which is much more flexible and convenient than the traditional solution that requires retraining the relevant model.

[0130] In one embodiment of this application, another image processing method is provided, which can be applied to Figure 1 The image processing system shown can be executed by a terminal device or a server, or by both. In this embodiment, the method is described using the server as an example. Figure 4 The diagram illustrates a flowchart of another image processing method. Figure 2 The method shown is an extension of the one presented.

[0131] The details of S401 to S405 are as follows:

[0132] S401. Perform text recognition processing on the virtual resource product image to obtain multiple text segments.

[0133] In the embodiments of this application, the specific implementation of text recognition processing can be found in step S201 regarding the specific implementation of text recognition processing, and will not be repeated here.

[0134] S402. If the text features corresponding to multiple consecutive text segments in multiple text segments are detected and match the data extraction types corresponding to the sub-rules in the information extraction rule sequence one by one, then the multiple consecutive text segments are regarded as multiple text segments that match the information extraction rule sequence.

[0135] In this embodiment of the application, the identified multiple text segments are arranged according to their positions in the virtual resource product image; at the same time, the sub-rules in the information extraction rule sequence can be arranged according to the positional order of the product description information used to describe the corresponding product attributes in the source application of the virtual resource product image.

[0136] The specific process of detecting whether multiple consecutive text segments match one-to-one with sub-rules in the information extraction rule sequence can include: First, determining the first sub-rule in the information extraction rule sequence as the target sub-rule; then, traversing the multiple identified text segments according to their order, and if the text features of the currently traversed text segment match the data extraction type corresponding to the target sub-rule, then writing the currently traversed text segment into the candidate text queue, and updating the target sub-rule to the next sub-rule in the information extraction rule sequence, until all the identified text segments have been traversed; finally, the multiple text segments in the candidate text queue can be considered as multiple consecutive text segments.

[0137] If the target sub-rule is the last sub-rule in the information extraction rule sequence, then the next sub-rule of the target sub-rule in the information extraction rule sequence can be determined as the first sub-rule in the information extraction rule sequence.

[0138] Optionally, if the text features of the currently traversed text segment do not match the data extraction type corresponding to the target sub-rule, and the target sub-rule is the first sub-rule, then traverse the next text segment; if the text features of the currently traversed text segment do not match the data extraction type corresponding to the target sub-rule, and the target sub-rule is not the first sub-rule, then the text segment written to the candidate text queue is discarded, and the target sub-rule is updated to the first sub-rule; wherein, the number of discarded text segments is the same as the number of sub-rules whose order precedes the target sub-rule, and the discarded text segments are the most recently written text segments to the candidate text queue (i.e., the text segment at the tail of the candidate text queue).

[0139] Furthermore, if the number of newly written text segments in the candidate text queue reaches the total number of sub-rules in the information extraction rule sequence, then it can be determined that the text features corresponding to multiple consecutive text segments in the detected text segments are matched one-to-one with the data extraction types corresponding to multiple consecutive sub-rules in the information extraction rule sequence.

[0140] In the specific implementation, see Figure 3 In the example shown, the first sub-rule in information extraction rule sequence 303 can be identified as the target sub-rule. The first line of text in text list 302, being a time number, does not match the data extraction type "string" of the target sub-rule, so the next text line is traversed. The second line of text, "small payment," matches the data extraction type of the first sub-rule, so it is added to the candidate text queue, and the target sub-rule is updated to the second sub-rule of information extraction rule sequence 303. The second line of text, "total assets (yuan)," does not match the data extraction type "floating-point number" of the second sub-rule, and since the target sub-rule is the second sub-rule, the most recently added text line to the candidate text queue, "small payment," is discarded, and the target sub-rule is updated to the first sub-rule of information extraction rule sequence 303. The matching process for lines 4 to 10 is similar and will not be elaborated here.

[0141] When traversing to the 11th line of text, the target sub-rule is the first sub-rule. The text features of the 11th line match the data extraction type corresponding to the first sub-rule, so the 11th line is added to the candidate text queue, and the target sub-rule is updated to the second sub-rule. Since the text features of the 12th line match the data extraction type corresponding to the second sub-rule, the 12th line is also added to the candidate text queue, and the target sub-rule is updated to the third sub-rule. The matching process for lines 12 to 14 is similar and will not be elaborated here.

[0142] After traversing all text segments in text list 302, the candidate text queue contains text segments from line 12 to line 18, where text segments from line 12 to line 14 are a text sequence; and text segments from line 15 to line 18 are a text sequence.

[0143] In one possible implementation, different applications may add different tags or other information to the virtual resource product information of the same product, or customize the abbreviation of the virtual resource product, leading to information identification errors in the virtual resource product information. For example, the product "JY Medical Health Hybrid Initiation A" may be called "JY Health" in one application, while in another application platform it may be tagged with "elderly care" and called "JY Medical Health Hybrid Initiation A (Elderly Care)".

[0144] Furthermore, different applications may use different data formats to represent virtual resource product information with the same product attributes. For example, some applications directly use numbers to represent the holding amount, while others use traditional Chinese characters to represent the holding amount.

[0145] Therefore, text segments containing a first specified character can be discarded based on the first specified character, and the data format of text segments containing a second specified character can be converted to a specified format to obtain updated text segments. Then, if the text features corresponding to consecutive text segments in the updated text segments are detected to match the data extraction types corresponding to the sub-rules in the information extraction rule sequence, then the consecutive text segments in the updated text segments are taken as the text segments that match the information extraction rule sequence.

[0146] The first specified character can be set based on tags or other information typically added to the virtual resource product information. It can also be information that easily causes text content clutter, such as blank lines. Optionally, considering that different applications add different information to the virtual resource product information, different first specified characters can be set for different applications.

[0147] Furthermore, the second specified character can be determined based on the virtual resource product information that may be expressed in different data formats depending on the application. The specified format can be manually set, or it can be set by the terminal device or server in the aforementioned image processing system, and is not limited here. Specifically, the specified format can be a data format that is easy to read and understand.

[0148] In one possible implementation, the virtual resource product image mentioned above may come from screenshots of virtual resource product interfaces in some applications; however, in cases where the wrong screenshot is uploaded, the virtual resource product image may not be a screenshot of the virtual resource product interface, but rather a screenshot of other interfaces in the application.

[0149] Therefore, in order to avoid wasting resources on extracting information from erroneous screenshots or images, the interface description keywords contained in the virtual resource product interface of the source application can also be obtained; if the identified multiple text segments contain text segments that match the interface description keywords, then step S402 is triggered; if the identified multiple text segments do not contain text segments that match the interface description keywords, then recognition failure information for the virtual resource product image is output.

[0150] The keywords for the interface description can be derived from fixed terms contained in the virtual resource product interface, such as... Figure 3In the example, the keyword for the interface description can be "total assets (RMB)". The keyword for the interface description may differ for different applications and different virtual resource products.

[0151] S403. Based on each sub-rule in the information extraction rule sequence, extract information from each matched text segment to obtain the product description information extracted by each sub-rule.

[0152] In this application embodiment, the specific implementation of step S403 can be found in the specific implementation of step S203, and will not be repeated here.

[0153] S404. By merging the product description information extracted from sub-rules corresponding to the same product attribute, the virtual resource product information contained in the virtual resource product image is obtained.

[0154] In this embodiment of the application, the virtual resource product information may include the product name of the virtual resource product. However, some product names may be too long, which may cause the product name of the same virtual resource product to be identified as two text segments, thus leading to errors in the extracted virtual resource product information.

[0155] Therefore, the information extraction rule sequence can include header name sub-rules and tail name sub-rules, and the header name sub-rules can be arranged in the tail name sub-rules; then, the information extraction process can include: extracting each character in the header text segment that matches the header name sub-rule based on the header name sub-rule, to obtain the header name description information extracted according to the header name sub-rule; extracting each character in the tail text segment that matches the tail name sub-rule based on the tail name sub-rule, to obtain the tail name description information extracted according to the tail name sub-rule; wherein, the header text segment and the tail text segment differ in at least one of the vertical horizontal coordinates and the horizontal vertical coordinates in the virtual resource product image.

[0156] Among them, based on the arrangement of various virtual resource product information in the source application of the virtual resource product image, other corresponding sub-rules can also be arranged between the header name sub-rule and the footer name sub-rule.

[0157] Therefore, during the merging process, the header name description information and the tail name description information can be concatenated sequentially according to the order of the header name sub-rules and the tail name sub-rules to obtain the product name of the virtual resource product in the virtual resource product image.

[0158] For example, see the second information extraction rule sequence for "Small Payment" in Table 1: "\n([^.+\n]+)\n([+-]?\d+\.\d+)\n([+-]?\d+\.\d+)\n(.*)\n([+-]?\d+\.\d+)\n([+-]?\d+\.\d+)%". The first sub-rule, "([^.+\n]+)", is the header name sub-rule, used to extract the first half of the fund name, and the fourth sub-rule, "n(.*)", is the tail name sub-rule, used to extract the second half of the fund name. Considering that if the fund name in the "small payment" application has a line break, the holding amount and holding profit will be arranged between the two parts of the fund name caused by the line break during text recognition; therefore, the second sub-rule in this information extraction rule sequence is used to extract the holding amount, the third sub-rule is used to extract the holding profit, the fifth sub-rule is used to extract yesterday's profit, and the sixth sub-rule is used to extract the holding rate of return.

[0159] It should be noted that multiple sub-rules can be set not only for cases where the product name information is truncated, but also for cases where other product attributes, such as holding amount, holding profit, and holding return rate, are truncated; this is not limited here. Furthermore, information of the same product attribute may be truncated into two or more text segments during text recognition. Correspondingly, the number of sub-rules in the information extraction rule sequence used to extract the product description information of that product attribute can also be two or more.

[0160] S405. Obtain the similarity between the product name of each virtual resource product in the publicly available product list and the product name in the virtual resource product information.

[0161] In this embodiment, different application platforms will launch a variety of virtual resource products, so virtual resource products from different applications can be aggregated into a public product list. Optionally, the product names of virtual resource products in the public product list can use common or standard names of virtual resource products, or they can use the naming name of the application that launched the virtual resource product.

[0162] The specific process of obtaining similarity may include: performing feature extraction processing on the product names of each virtual resource product in the publicly available product list to obtain a first feature vector; and performing feature extraction processing on the product names contained in the virtual resource product information to obtain a second feature vector; calculating the vector distance between the first feature vector and the second feature vector; and obtaining the similarity between the product names of each virtual resource product and the product names in the virtual resource product information based on the vector distance.

[0163] Alternatively, one can obtain the similarity between the product name of each virtual resource product in the public product list and the product name in the virtual resource product information through other methods used to obtain the similarity between two texts, which will not be elaborated here.

[0164] S406. Add the publicly available product information corresponding to the virtual resource product with the highest similarity to the product management application.

[0165] Because the product name in the virtual resource product image may not be the more common name of the corresponding virtual resource product, different applications may have different custom abbreviations for the same virtual resource product. This will cause the product management application to be unable to add the correct virtual resource product after recognizing the product name in the virtual resource product image. On the other hand, establishing the mapping relationship between virtual resource products and various custom abbreviations is too cumbersome and requires frequent maintenance of the mapping relationship as the custom abbreviations change.

[0166] Therefore, in this embodiment of the application, the virtual resource product with the highest similarity in the publicly available product list will be identified as the virtual resource product contained in the virtual resource product image, and its corresponding public product information will be added to the product management application.

[0167] The publicly available information about virtual resource products can come from the source application's explanation of the virtual resource product, or from other platforms or websites; there are no restrictions on this.

[0168] Optionally, other information contained in the extracted virtual resource product information can also be added to the product management application.

[0169] For specific implementation details, please refer to the appendix. Figure 5 The diagram illustrates a strategy for extracting virtual resource product information. This strategy may involve three main entities: an object, a policy server, and an OCR (Optical Character Recognition) system.

[0170] First, as shown in step S501, the object that needs to extract virtual resource product information from the image can upload the image after selecting the application and business type. Here, the selected application refers to the source application of the image, while the business type is used to characterize the virtual resource product interface contained in the uploaded image.

[0171] For practical applications, please refer to the appendix. Figure 6 This illustrates a schematic diagram of the management interface for a product management application. For example... Figure 6As shown, the management interface 601 of the product management application "Xiao F Fund Management" contains virtual resource products from applications such as "Xiao Pay" and "KK". At the same time, for each virtual resource product, the product name, holding amount, holding income, holding yield, and today's income are also clearly marked.

[0172] When an object wants to add a new virtual resource product to the product management application "Xiao F Yangji", please refer to the appendix. Figure 7 The diagram illustrates an image upload interface. Figure 7 As shown, in the image upload interface 701, the source application of the image the object wants to add can be selected, such as "XiaoPay", "KK", "Wealth Management T", "XQ", "XX Finance", etc. Optionally, the object can also select the "Other" option. For applications in the "Other" option, there may not be a pre-set corresponding information extraction rule sequence. In this case, the information extraction rule sequence corresponding to the image with the highest similarity to the object's uploaded image can be found from historically processed images and used as the information extraction rule sequence for images from applications in the "Other" option. Optionally, rule configuration prompts can also be sent to the staff of the product management application so that they can configure the corresponding information extraction rule sequence in a timely manner.

[0173] Furthermore, after selecting an application, as shown in image upload interface 701, screenshot prompts for the selected application are displayed to prevent users from taking incorrect screenshots, thus improving the efficiency of extracting information about virtual resource products. Additionally, users can choose between manual addition and screenshot addition, both of which allow adding multiple images at once. Optionally, after a user clicks on manual addition or screenshot addition, they can be directly redirected to the virtual resource product interface for the selected application. This further reduces the risk of taking incorrect screenshots and improves the efficiency of extracting information about virtual resource products.

[0174] Secondly, as shown in steps 502 and 503, after acquiring the image, the policy server can call OCR to perform text recognition on the image. Then, as shown in steps 504 and 505, the OCR text recognition will return a text list. If the text list is empty, it indicates that the recognition failed, and the virtual resource product information extraction policy process can be terminated. If the text list is not empty, it indicates that the recognition was successful, and the application rules can be loaded. These application rules may include the information extraction rule sequence corresponding to the source application of the uploaded image, interface description keywords, and the first and second specified characters corresponding to the source application of the uploaded image.

[0175] Then, as shown in step 506, after the application rules are loaded, it can be further detected whether the uploaded image contains interface description keywords; if the image does not contain interface description keywords, it can be determined that the screenshot is incorrect, thus ending the virtual resource product information extraction strategy process. Furthermore, recognition failure information for the image can also be output, which may include image error information.

[0176] If the image does not contain the interface description keywords, as shown in step 507, the text segments in the text list obtained by text recognition can be further cleaned using the first specified character and the second specified character loaded in step 505 to obtain an updated text list.

[0177] Then, as shown in step 508, the information extraction rule sequence in the information extraction rule sequence list (composed of multiple information extraction rule sequences corresponding to the source application of the uploaded image) can be traversed in turn to find multiple texts in the updated text list that match each information extraction rule sequence, thereby extracting virtual resource product information from multiple matching text segments based on each information extraction rule sequence.

[0178] After extracting the virtual resource product information, as shown in step 510, the virtual resource product information can be further processed. For example, for information extraction rule sequences containing header name sub-rules and tail name sub-rules, the information extracted from the header name sub-rules can be merged with the information extracted from the tail name sub-rules.

[0179] Next, as shown in step 512, the similarity between the product names in the processed virtual resource product information and the product names of each virtual resource product in the publicly available product list can be obtained, thus generating a name similarity product list. This name similarity product list may include virtual resource products with a corresponding similarity greater than a preset similarity. The preset similarity can be manually set or set by the terminal device or server in the aforementioned image processing system; no limitation is made here.

[0180] Finally, as shown in step 513, the publicly available product information corresponding to the virtual resource product with the highest similarity can be obtained and added to the virtual resource product information. Furthermore, the completed virtual resource product information can be added to the product management application.

[0181] In general, the virtual resource product information extraction strategy divides the information extraction process into OCR recognition, text cleaning, text content discovery, and name matching. Steps 501 to 504 pertain to OCR recognition, which extracts the original text from the image and forms a text list. Steps 505 to 507 pertain to text cleaning, which mainly includes applying rules such as loading interface description keywords, first specified characters, and second specified characters. These rules are used to determine whether the virtual resource product image is the correct interface image, remove redundant data from the text list using the first specified character, and convert the data format to obtain formatted text data. Specifically, the information obtained from images is usually unstructured, containing a large amount of redundant data and various data formats, making it inconvenient for subsequent unified information extraction. Therefore, to improve information extraction efficiency, the text cleaning part is needed to clean up redundant data and perform data structuring.

[0182] The text content discovery section includes steps 508 to 510, which is the process of accurately extracting virtual resource product information from the text based on the information extraction rule sequence. Information extraction using the information extraction rule sequence effectively avoids false positives and false negatives. The name matching section includes steps 511 to 513, which is the process of finding the virtual resource product with the highest similarity to the product name contained in the virtual resource product information. Furthermore, the information extraction rule sequence in the virtual resource product information extraction strategy is configurable, allowing for configurable extensions. When supporting the extraction of virtual resource product information from images for completely different applications, no code modification is required; only rule configuration adjustments are needed.

[0183] For practical applications, please refer to the appendix. Figure 8 This illustrates a schematic diagram of a virtual resource product image. (Continued) Figure 7 As shown in the example, after selecting the "Wealth Management T" application, the object can take a screenshot of the virtual resource product interface in the "Wealth Management T" application to obtain the virtual resource product image 801, and then upload the virtual resource product image 801 to the "Little F Fund Management" application.

[0184] Please see the appendix next. Figure 9 The diagram illustrates a successful product addition screen. The application server of the "Little F Foundation" application uses a method such as... Figure 5 After the virtual resource product information extraction strategy shown extracts the virtual resource product information from virtual resource product image 801, the extracted virtual resource product information of each virtual resource product can be added to the "Little F Foundation" application. For example... Figure 9As shown in the product addition success interface 901, the product addition success interface 901 contains information such as the product name, holding amount, and holding profit of each virtual resource product in the virtual resource product image 801. The product addition success interface 901 may also include product codes and other public product information of each virtual resource product.

[0185] Some virtual resource products have rather long names, and different applications may use custom abbreviations to improve readability when listing virtual resource products. Therefore, the product names of the same virtual resource product may differ across applications. This means that even if the accurate virtual resource product information is identified and extracted, it may still be identified as a different virtual resource product because the product name in the information is a custom abbreviation used by the application, resulting in the incorrect virtual resource product being added.

[0186] In this embodiment, the similarity between the product names of each virtual resource product in the publicly available product list and the product names in the virtual resource product information is obtained, thereby finding the virtual resource product with the highest similarity to the product names in the virtual resource product information. The custom abbreviations of virtual resource products are derived from their generic or standard names, making them highly similar. Therefore, by finding the virtual resource product with the highest similarity, this embodiment effectively avoids the problem of low matching rates and finds more accurate virtual resource products. More accurate virtual resource products lead to more accurate publicly available product information, which is beneficial for adding more accurate virtual resource product information to product management applications, thereby improving management efficiency and enhancing the user experience.

[0187] Furthermore, considering that some virtual resource product names may contain text wrapping or column breaks due to their length, this embodiment also introduces header name sub-rules and tail name sub-rules into the information extraction rule sequence. This ensures that even when product names in the image contain line breaks, the accurate and complete product names can still be extracted, which helps improve the accuracy of virtual resource product information extraction and achieves the goal of extracting more accurate virtual resource product information. Simultaneously, this embodiment also uses methods such as redundant text deletion and text formatting to avoid interference caused by redundant information or inconsistent formats, which helps improve the efficiency of subsequent text segment processing and the accuracy of virtual resource product information extraction.

[0188] This application describes an apparatus embodiment that can be used to perform the image processing method described above. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the image processing method described above.

[0189] This application provides an image processing apparatus, such as... Figure 10 As shown, the device includes a text recognition unit 1001, a rule matching unit 1002, and an information extraction unit 1003, wherein:

[0190] The text recognition unit 1001 is used to perform text recognition processing on the virtual resource product image to obtain multiple text segments; wherein, the multiple text segments are arranged according to the position of each text segment in the virtual resource product image;

[0191] The rule matching unit 1002 is used to treat the consecutive text segments as multiple text segments that match the data extraction types corresponding to the sub-rules in the information extraction rule sequence if the text features corresponding to the consecutive text segments in the multiple text segments are detected. Among them, the product description information extracted by a sub-rule is used to describe a product attribute.

[0192] The information extraction unit 1003 is used to extract information from each matched text segment based on each sub-rule in the information extraction rule sequence, and to obtain the product description information extracted by each sub-rule.

[0193] The information extraction unit 1003 is also used to merge the product description information extracted by the sub-rules corresponding to the same product attribute to obtain the virtual resource product information contained in the virtual resource product image.

[0194] In one embodiment of this application, the information extraction rule sequence includes header name sub-rules and tail name sub-rules, with the header name sub-rules preceding the tail name sub-rules. Based on the aforementioned scheme, when the information extraction unit 1003 extracts information from each matched text segment based on each sub-rule in the information extraction rule sequence to obtain the product description information extracted by each sub-rule, it can specifically be used to extract each character in the header text segment that matches the header name sub-rule based on the header name sub-rule to obtain the header name description information extracted by the header name sub-rule; and to extract each character in the tail text segment that matches the tail name sub-rule based on the tail name sub-rule to obtain the tail name description information extracted by the tail name sub-rule; wherein, the header text segment and the tail text segment differ in at least one of the vertical horizontal coordinates and the horizontal vertical coordinates in the virtual resource product image. When the information extraction unit 1003 merges the product description information extracted by the sub-rules corresponding to the same product attribute to obtain the virtual resource product information contained in the virtual resource product image, it can specifically be used to sequentially concatenate the header name description information and the tail name description information according to the order of the header name sub-rules and the tail name sub-rules to obtain the product name of the virtual resource product in the virtual resource product image.

[0195] In one embodiment of this application, the identified multiple text segments are arranged according to their positions in the virtual resource product image. Based on the aforementioned scheme, the rule matching unit 1002 can also be used to determine the first sub-rule in the information extraction rule sequence as the target sub-rule. The identified multiple text segments are traversed according to their arrangement order. If the text features of the currently traversed text segment match the data extraction type corresponding to the target sub-rule, the currently traversed text segment is written into the candidate text queue, and the target sub-rule is updated to the next sub-rule in the information extraction rule sequence until the identified multiple text segments are traversed. The multiple text segments in the candidate text queue are treated as consecutive multiple text segments.

[0196] In one embodiment of this application, based on the foregoing scheme, the rule matching unit 1002 can be further configured to: if the text features of the currently traversed text segment do not match the data extraction type corresponding to the target sub-rule, and the target sub-rule is the first sub-rule, then traverse the next text segment; if the text features of the currently traversed text segment do not match the data extraction type corresponding to the target sub-rule, and the target sub-rule is not the first sub-rule, then discard the text segment written to the candidate text queue and update the target sub-rule to the first sub-rule; wherein, the number of discarded text segments is the same as the number of sub-rules whose order precedes the target sub-rule.

[0197] In one embodiment of this application, the virtual resource product image includes virtual resource product information of multiple virtual resource products. Based on the aforementioned scheme, when the information extraction unit 1003 extracts information from each matched text segment based on each sub-rule in the information extraction rule sequence to obtain the product description information extracted by each sub-rule, it can specifically be used to: take the consecutive multiple text segments detected each time from the multiple text segments that match the information extraction rule sequence as the text group corresponding to each virtual resource product; extract information from the text group corresponding to each virtual resource product based on the information extraction rule sequence to obtain the virtual resource product information of each virtual resource product.

[0198] In one embodiment of this application, the number of information extraction rule sequences includes multiple sequences. Based on the aforementioned scheme, before the rule matching unit 1002 detects that the text features corresponding to multiple consecutive text segments in multiple text segments match the data extraction types corresponding to the sub-rules in the information extraction rule sequence, it can also be used to: obtain the application identifier of the source application of the virtual resource product image; and search for the target information extraction rule sequence corresponding to the obtained application identifier according to the pre-set correspondence between the application identifier and the information extraction rule sequence, so as to extract the virtual resource product information contained in the virtual resource product image based on the target information extraction rule sequence.

[0199] In one embodiment of this application, based on the foregoing scheme, the text recognition unit 1001 can also be used to: obtain interface description keywords contained in the virtual resource product interface of the source application; if the recognized multiple text segments contain text segments that match the interface description keywords, then trigger the execution of the step of if the text features corresponding to the multiple consecutive text segments in the multiple text segments are detected to match the data extraction types corresponding to the sub-rules in the information extraction rule sequence, then treat the multiple consecutive text segments as multiple text segments that match the information extraction rule sequence; if the recognized multiple text segments do not contain text segments that match the interface description keywords, then output recognition failure information for the virtual resource product image.

[0200] In one embodiment of this application, the virtual resource product information includes the product name of the virtual resource product; based on the aforementioned scheme, the information extraction unit 1003 can also be used to: obtain the similarity between the product name of each virtual resource product in the publicly available product list and the product name in the virtual resource product information, wherein the virtual resource products in the publicly available product list come from different applications; and add the publicly available product information corresponding to the virtual resource product with the highest similarity to the product management application.

[0201] In one embodiment of this application, based on the aforementioned scheme, when the rule matching unit 1002 detects that the text features corresponding to multiple consecutive text segments in a plurality of text segments match the data extraction types corresponding to the sub-rules in the information extraction rule sequence, the multiple consecutive text segments are used as multiple text segments that match the information extraction rule sequence. Specifically, this can be done by: discarding text segments containing a first specified character from the multiple text segments obtained by text recognition, and converting the data format of text segments containing a second specified character from the multiple text segments obtained by text recognition into a specified format to obtain updated multiple text segments; if the text features corresponding to multiple consecutive text segments in the updated multiple text segments match the data extraction types corresponding to the multiple consecutive sub-rules in the information extraction rule sequence, the multiple consecutive text segments are used as multiple text segments that match the information extraction rule sequence.

[0202] It should be noted that the apparatus provided in the above embodiments and the method provided in the above embodiments belong to the same concept, and the specific way in which each module and unit performs operations has been described in detail in the method embodiments, and will not be repeated here.

[0203] The apparatus provided in the above embodiments can be located within a terminal device or a server. The apparatus provided in this application filters text segments containing information unrelated to virtual resource products by finding multiple text segments that match the information extraction rule sequence. This avoids the impact of text segments containing invalid information on the information extraction process, thus improving the accuracy of information extraction. Furthermore, considering that virtual resource product information appears continuously in an image, detecting whether multiple consecutive text segments match multiple consecutive sub-rules effectively avoids information extraction errors caused by changes in the arrangement format of virtual resource product information in the image, leading to more accurate extraction of virtual resource product information. Finally, this solution further refines information extraction by merging product description information corresponding to the same product attribute, resulting in even more accurate extraction of virtual resource product information.

[0204] Embodiments of this application also provide an electronic device, including one or more processors and a storage device, wherein the storage device is used to store one or more computer programs, which, when executed by one or more processors, cause the electronic device to implement the image processing method described above.

[0205] Figure 11 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown.

[0206] It should be noted that, Figure 11 The computer system 1100 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0207] like Figure 11 As shown, the computer system 1100 includes a central processing unit (CPU) 1101, which can perform various appropriate actions and processes, such as executing the methods described in the above embodiments, based on a program stored in read-only memory (ROM) 1102 or a program loaded from storage portion 1108 into random access memory (RAM) 1103. The RAM 1103 also stores various programs and data required for system operation. The CPU 1101, ROM 1102, and RAM 1103 are interconnected via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0208] In some embodiments, the following components are connected to I / O interface 1105: an input section 1106 including a keyboard, mouse, etc.; an output section 1107 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1108 including a hard disk, etc.; and a communication section 1109 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 1109 performs communication processing via a network such as the Internet. A drive 1110 is also connected to I / O interface 1105 as needed. A removable medium 1111, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 1110 as needed so that computer programs read from it can be installed into storage section 1108 as needed.

[0209] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1109, and / or installed from removable medium 1111. When the computer program is executed by processor (CPU) 1101, it performs various functions defined in the system of this application.

[0210] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory, flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0211] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and a computer program.

[0212] The units or modules described in the embodiments of this application can be implemented in software or hardware, and can also be located in a processor. The names of these units or modules do not necessarily limit the specific unit or module itself.

[0213] Another aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the image processing method as described above. This computer-readable storage medium may be included in the electronic device described in the above embodiments, or it may exist independently and not incorporated into the electronic device.

[0214] Another aspect of this application provides a computer program product comprising a computer program stored in a computer-readable storage medium. A processor of an electronic device reads the computer program from the computer-readable storage medium and executes the computer program, causing the electronic device to perform the image processing method as described above in the various embodiments.

[0215] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0216] Other embodiments of this application will readily conceive of by considering the specification and practicing the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0217] The above content is merely a preferred exemplary embodiment of this application and is not intended to limit the implementation of this application. Those skilled in the art can easily make corresponding modifications or alterations based on the main concept and spirit of this application. Therefore, the scope of protection of this application should be determined by the scope of protection claimed in the claims.

Claims

1. An image processing method, characterized in that, The method includes: The virtual resource product image is subjected to text recognition processing to obtain multiple text segments; wherein, the multiple text segments are arranged according to their positions in the virtual resource product image; If the text features corresponding to multiple consecutive text segments in the multiple text segments are detected to match the data extraction types corresponding to the sub-rules in the information extraction rule sequence, then the multiple consecutive text segments are regarded as multiple text segments that match the information extraction rule sequence; wherein, the product description information extracted by a sub-rule is used to describe a product attribute. Based on each sub-rule in the information extraction rule sequence, information is extracted from each matched text segment to obtain the product description information extracted by each sub-rule. By merging the product description information extracted from sub-rules corresponding to the same product attribute, the virtual resource product information contained in the virtual resource product image is obtained.

2. The method according to claim 1, characterized in that, The information extraction rule sequence includes header name sub-rules and footer name sub-rules, with the header name sub-rules preceding the footer name sub-rules. The information extraction is performed on each matched text segment based on each sub-rule in the information extraction rule sequence to obtain the product description information extracted by each sub-rule, including: Based on the header name sub-rule, extract each character from the header text segment that matches the header name sub-rule to obtain the header name description information extracted by the header name sub-rule; Based on the tail name sub-rule, each character in the tail text segment that matches the tail name sub-rule is extracted to obtain the tail name description information extracted by the tail name sub-rule; wherein, the head text segment and the tail text segment differ in at least one of the vertical horizontal coordinates and the horizontal vertical coordinates in the virtual resource product image. The process of merging the product description information extracted from sub-rules corresponding to the same product attribute to obtain the virtual resource product information contained in the virtual resource product image includes: According to the order of the header name sub-rules and the tail name sub-rules, the header name description information and the tail name description information are concatenated sequentially to obtain the product name of the virtual resource product in the virtual resource product image.

3. The method according to claim 1, characterized in that, The method further includes: The first sub-rule in the information extraction rule sequence is identified as the target sub-rule; The multiple text segments are traversed in the order of their identification. If the text features of the currently traversed text segment match the data extraction type corresponding to the target sub-rule, the currently traversed text segment is written into the candidate text queue, and the target sub-rule is updated to the next sub-rule in the information extraction rule sequence, until all the multiple text segments have been traversed. Multiple text segments in the candidate text queue are taken as the consecutive multiple text segments.

4. The method according to claim 2, characterized in that, The method further includes: If the text features of the currently traversed text segment do not match the data extraction type corresponding to the target sub-rule, and the target sub-rule is the first sub-rule, then traverse the next text segment; If the text features of the currently traversed text segment do not match the data extraction type corresponding to the target sub-rule, and the target sub-rule is not the first sub-rule, then the text segment written into the candidate text queue is discarded, and the target sub-rule is updated to the first sub-rule; wherein, the number of discarded text segments is the same as the number of sub-rules arranged before the target sub-rule.

5. The method according to claim 1, characterized in that, The virtual resource product image includes virtual resource product information for multiple virtual resource products; the information extraction is performed on each matched text segment based on each sub-rule in the information extraction rule sequence to obtain the product description information extracted according to each sub-rule, including: The consecutive text segments detected each time from multiple text segments that match the information extraction rule sequence are taken as the text sequence corresponding to each virtual resource product. Based on the information extraction rule sequence, information is extracted from the text sequence corresponding to each virtual resource product to obtain the virtual resource product information of each virtual resource product.

6. The method according to claim 1, characterized in that, The number of information extraction rule sequences includes multiple sequences; before the step of determining whether consecutive text segments in the multiple text segments are matched one-to-one with the data extraction types corresponding to the sub-rules in the information extraction rule sequence, the method further includes: Obtain the application identifier of the source application of the virtual resource product image; According to the pre-defined correspondence between application identifiers and information extraction rule sequences, the target information extraction rule sequence corresponding to the obtained application identifier is searched, and the virtual resource product information contained in the virtual resource product image is extracted based on the target information extraction rule sequence.

7. The method according to claim 6, characterized in that, The method further includes: Obtain the interface description keywords contained in the virtual resource product interface of the source application; If the identified multiple text segments contain text segments that match the keywords of the interface description, then the step of "if the text features corresponding to multiple consecutive text segments in the multiple text segments are detected and matched one by one with the data extraction types corresponding to the sub-rules in the information extraction rule sequence, then the multiple consecutive text segments are regarded as multiple text segments that match the information extraction rule sequence" is triggered. If none of the identified text segments contains a text segment that matches the interface description keywords, then a recognition failure message for the virtual resource product image is output.

8. The method according to any one of claims 1 to 7, characterized in that, The virtual resource product information includes the product name of the virtual resource product; the method further includes: Obtain the similarity between the product name of each virtual resource product in the publicly available product list and the product name in the virtual resource product information, wherein the virtual resource products in the publicly available product list come from different applications; Add the publicly available product information corresponding to the virtual resource product with the highest similarity to the product management application.

9. The method according to any one of claims 1 to 7, characterized in that, If the text features corresponding to multiple consecutive text segments in the plurality of text segments are detected to match the data extraction types corresponding to the sub-rules in the information extraction rule sequence, then the multiple consecutive text segments are regarded as multiple text segments that match the information extraction rule sequence, including: For text segments containing a first specified character obtained from multiple text segments obtained from text recognition, a discard operation is performed on the first specified character, and the data format of text segments containing a second specified character obtained from multiple text segments obtained from text recognition is converted to a specified format to obtain updated multiple text segments. If the text features corresponding to multiple consecutive text segments in the updated multiple text segments are detected to match the data extraction types corresponding to the sub-rules in the information extraction rule sequence, then the multiple consecutive text segments are regarded as multiple text segments that match the information extraction rule sequence.

10. An image processing apparatus, characterized in that, The device includes a text recognition unit, a rule matching unit, and an information extraction unit, wherein: The text recognition unit is used to perform text recognition processing on virtual resource product images to obtain multiple text segments; The rule matching unit is configured to, if it detects that the text features corresponding to multiple consecutive text segments in the multiple text segments match the data extraction types corresponding to the sub-rules in the information extraction rule sequence, then regard the multiple consecutive text segments as multiple text segments that match the information extraction rule sequence; wherein, the product description information extracted by a sub-rule is used to describe a product attribute. The information extraction unit is used to extract information from each matched text segment based on each sub-rule in the information extraction rule sequence, and to obtain the product description information extracted by each sub-rule. The information extraction unit is further configured to merge the product description information extracted by sub-rules corresponding to the same product attribute to obtain the virtual resource product information contained in the virtual resource product image.

11. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the image processing method as described in any one of claims 1 to 9.

12. An electronic device, characterized in that, include: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the image processing method as described in any one of claims 1 to 9.

Images