Semantic information recognition method, device, electronic device and storage medium

Abstract

The present invention discloses a semantic information recognition method, device, electronic device and storage medium. The method comprises: obtaining an image to be recognized; the image to be recognized includes text; performing visual recognition processing on the image to be recognized to obtain visual feature information of the image to be recognized; performing character position alignment processing on the visual feature information to obtain a character information sequence corresponding to the text; wherein the position of the character information of each word in the text in the character information sequence is the same as the position of each word in the text; and determining the semantic information of the text based on the character information of each word and the character information of the associated words of each word in the character information sequence. By obtaining the semantic information of the text using the character information of each word and the character information of the associated words in the image containing text, the error of the latter solution caused by the error of the former solution due to the fact that the text recognition and semantic extraction are two independent solutions can be avoided, thereby improving the recognition accuracy.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a semantic information recognition method, device, electronic device and storage medium. Background Art

[0002] With the development of Internet technology, there are more and more opportunities for text image applications in various fields, such as outdoor billboards, signage, faxes, and book scans. This makes it increasingly important to extract text and understand its semantics, as in the examples above.

[0003] Related technologies primarily utilize OCR (optical character recognition) technology to identify text images and obtain the text within them. Semantic extraction is then performed based on the obtained text. However, when errors (such as typos) are present in the text obtained by OCR, this inevitably affects subsequent semantic extraction of the text, reducing recognition accuracy. Summary of the Invention

[0004] In order to solve the problems of the prior art, the embodiments of the present invention provide a semantic information recognition method, device, electronic device and storage medium. The technical solution is as follows:

[0005] In one aspect, a method for identifying semantic information is provided, the method comprising:

[0006] Acquire an image to be recognized; the image to be recognized includes text;

[0007] Performing visual recognition processing on the image to be identified to obtain visual feature information of the image to be identified;

[0008] Performing character position alignment processing on the visual feature information to obtain a character information sequence corresponding to the text; wherein the position of the character information of each word in the text in the character information sequence is the same as the position of each word in the text; the visual feature information includes at least one of edge feature information, corner feature information, color feature information, texture feature information, and shape feature information of the image to be recognized;

[0009] The semantic information of the text is determined based on the character information of each word and the character information of the associated words of each word in the character information sequence.

[0010] In another aspect, a semantic information recognition device is provided, comprising:

[0011] An image acquisition module is used to acquire an image to be recognized, wherein the image to be recognized includes text;

[0012] a visual processing module, configured to perform visual recognition processing on the image to be recognized to obtain visual feature information of the image to be recognized; the visual feature information includes at least one of edge feature information, corner feature information, color feature information, texture feature information, and shape feature information of the image to be recognized;

[0013] An alignment processing module is used to perform character position alignment processing on the visual feature information to obtain a character information sequence corresponding to the text; wherein the position of the character information of each word in the text in the character information sequence is the same as the position of each word in the text;

[0014] The semantic determination module is used to determine the semantic information of the text according to the character information of each word and the character information of the associated words of each word in the character information sequence.

[0015] In some possible embodiments,

[0016] A visual processing module is used to perform visual recognition processing on the image to be recognized based on the visual feature recognition module in the target semantic recognition network to obtain visual feature information of the image to be recognized;

[0017] An alignment processing module is used to perform character position alignment processing on the visual feature information based on the position alignment module in the target semantic recognition network to obtain a character information sequence corresponding to the text;

[0018] The semantic determination module is used to perform semantic recognition on the character information of each word and the character information of the associated words of each word in the character information sequence based on the character semantic recognition module in the target semantic recognition network to obtain the semantic information of the text.

[0019] In some possible embodiments, the method further includes a network training module for:

[0020] Acquire a sample image set; the sample image set includes a plurality of sample images and preset semantic information corresponding to each sample image; the plurality of sample images have the same size; each of the plurality of sample images includes sample text;

[0021] The visual feature recognition module in the original semantic recognition network performs visual recognition processing on the sample image set to obtain the visual feature information of each sample image; the visual feature information of each sample image is one-dimensional information;

[0022] Based on the position alignment module in the original semantic recognition network, the visual feature information of each sample image is aligned with the character position to obtain the character information sequence corresponding to the sample text contained in each sample image;

[0023] Based on the mask module in the original semantic recognition network, the character information sequence corresponding to each sample image is masked to obtain the masked character information sequence corresponding to each sample image;

[0024] Based on the character semantic recognition module in the original semantic recognition network, semantic recognition is performed on the mask character information sequence corresponding to each sample image to obtain the predicted semantic information corresponding to each sample image;

[0025] The original semantic recognition network is trained based on the predicted semantic information corresponding to each sample image and the preset semantic information corresponding to each sample image to obtain the target semantic recognition network.

[0026] In some possible embodiments, the network training module is used to:

[0027] Based on the position alignment module in the original semantic recognition network, the sub-visual feature information corresponding to each word in the sample text included in each sample image is classified into character information of each word;

[0028] Determining a position identifier of the character information of each word based on the position of the sub-visual feature information corresponding to each word in the visual feature information of each sample image;

[0029] Determining a character information sequence corresponding to a sample text contained in each sample image based on the character information of each word and a position identifier of the character information of each word;

[0030] The position of the character information of each word in the sample text in the corresponding character information sequence of the sample text is the same as the position of each word in the sample text.

[0031] In some possible embodiments, the network training module is used to:

[0032] Adding spacing character information after the character information sequence corresponding to each sample image to obtain a first information sequence corresponding to each sample image;

[0033] In the sample image set, N first information sequences corresponding to N sample images are spliced ​​together to obtain a second information sequence, where N is a positive integer greater than 1;

[0034] Based on the mask module in the original semantic recognition network, character information masking processing is performed on the first information sequence and the second information sequence respectively to obtain a mask character information sequence set corresponding to the sample image set.

[0035] In some possible embodiments, the network training module is used to:

[0036] Performing single-word character information masking processing on the first information sequence based on the mask module in the original semantic recognition network to obtain a first masked information sequence;

[0037] performing single-word character information masking and continuous multi-word character information masking processing on the second information sequence based on the masking module in the original semantic recognition network to obtain a second masked information sequence;

[0038] Determining a mask character information sequence set corresponding to the sample image set based on the first mask information sequence and the second mask information sequence;

[0039] The proportion of mask characters in the first mask information sequence and the second mask information sequence is less than or equal to a preset value.

[0040] In some possible embodiments, the network training module is used to:

[0041] Determine a semantic loss function based on the predicted semantic information corresponding to each sample image and the preset semantic information corresponding to each sample image;

[0042] Based on the semantic loss function, the network parameters of the original semantic recognition network are updated to obtain an updated original semantic recognition network;

[0043] The original semantic recognition network is trained cyclically until the iteration termination condition is met;

[0044] The target semantic recognition network is obtained by combining the visual feature recognition module, position alignment module and character semantic recognition module in the trained original semantic recognition network.

[0045] In some possible embodiments, the network training module is used to:

[0046] Acquire a plurality of original images; each of the plurality of original images includes sample text;

[0047] Performing text position detection on multiple original images to obtain position information of sample text in each original image;

[0048] Cropping each original image based on position information of the sample text in each original image to obtain a cropped image containing the sample text;

[0049] An image size conversion process is performed on each cropped image to obtain a sample image set; each sample image in the sample image set is an image with the same size after the image size conversion.

[0050] On the other hand, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the above-mentioned semantic information recognition method.

[0051] On the other hand, a computer-readable storage medium is provided, in which at least one instruction or at least one program is stored. The at least one instruction or the at least one program is loaded and executed by a processor to implement the semantic information recognition method as described above.

[0052] In another aspect, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the above-described semantic information recognition method.

[0053] The embodiment of the present invention obtains an image to be recognized; the image to be recognized includes text; performs visual recognition processing on the image to be recognized to obtain visual feature information of the image to be recognized; the visual feature information includes at least one of edge feature information, corner feature information, color feature information, texture feature information and shape feature information of the image to be recognized; performs character position alignment processing on the visual feature information to obtain a character information sequence corresponding to the text; wherein the position of the character information of each word in the text in the character information sequence is the same as the position of each word in the text; and determines the semantic information of the text based on the character information of each word and the character information of the associated words of each word in the character information sequence. The embodiment of the present application can obtain the semantic information of the text from an image containing text by using a complete technology, using the character information of each word and the related words of the word, and can avoid the error of the latter scheme caused by the error of the former scheme due to the fact that the text recognition and semantic extraction are two independent schemes to the greatest extent possible, thereby improving the recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0055] Figure 1 is a schematic diagram of an implementation environment provided by an embodiment of the present invention;

[0056] Figure 21 is a flow chart of a semantic information recognition method provided by an embodiment of the present invention;

[0057] Figure 3 1 is a schematic diagram of the structure of a trained target semantic recognition network provided by an embodiment of the present invention;

[0058] Figure 4 is a structural diagram of a residual structure provided by an embodiment of the present invention;

[0059] Figure 5 This is a schematic diagram of a process for determining a character information sequence corresponding to a text provided by an embodiment of the present invention;

[0060] Figure 6 This is a flow chart of a training target semantic recognition network provided by an embodiment of the present invention;

[0061] Figure 7 This is a structural block diagram of a semantic information recognition device provided by an embodiment of the present invention;

[0062] Figure 8 This is a hardware structure block diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0063] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0064] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way are interchangeable where appropriate so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0065] It is understandable that in the specific implementation of this application, related data such as user information is involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0066] In order to facilitate understanding of the above-mentioned technical solutions and the technical effects produced by the embodiments of the present disclosure, a brief introduction is given to the terms involved in the embodiments of the present disclosure:

[0067] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also involves studying the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0068] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0069] Computer vision (CV) is the science of making machines "see." Specifically, it refers to machine vision, where cameras and computers replace the human eye in identifying and measuring objects, performing further image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0070] Natural language processing (NLP) is a key area of ​​research in computer science and artificial intelligence. It studies the theories and methods that enable effective communication between humans and computers using natural language. Natural language processing (NLP) integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language we use in everyday life—and is closely linked to the study of linguistics. Natural language processing technologies typically include text processing, semantic understanding, machine translation, robotic question answering, and knowledge graphs.

[0071] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0072] Autonomous driving technology usually includes high-precision maps, environmental perception, behavioral decision-making, path planning, motion control and other technologies. Autonomous driving technology has broad application prospects.

[0073] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, autonomous driving, drones, robots, smart medical care, smart customer service, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0074] The embodiments of the present invention can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, etc.

[0075] Cloud technology refers to a hosting technology that unifies hardware, software, and network resources within a wide area network (WAN) or local area network (LAN) to enable data computing, storage, processing, and sharing. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, and application technology based on the cloud computing business model. It can form a resource pool that can be used on demand with flexibility and convenience. Cloud computing technology will become a crucial support. Backend services for technical network systems, such as video websites, image websites, and more portals, require extensive computing and storage resources. With the rapid development and application of the internet industry, every item will likely have its own unique identification mark and will need to be transmitted to backend systems for logical processing. Data of varying levels will be processed separately, and data from all industries will require a strong system backend, which can only be achieved through cloud computing.

[0076] Cloud storage is a new concept that has evolved from the concept of cloud computing. A distributed cloud storage system (hereafter referred to as a storage system) is a storage system that uses cluster applications, grid technology, and distributed storage file systems to bring together a large number of different types of storage devices (also known as storage nodes) on a network through application software or application interfaces to work together and provide data storage and service access. Currently, storage systems create logical volumes. When creating a logical volume, physical storage space is allocated for each logical volume. This physical storage space may consist of disks on a single storage device or several storage devices. When a client stores data on a logical volume, it stores the data on a file system. The file system divides the data into multiple parts, each of which is an object. An object contains not only the data but also additional information such as the data identifier (ID, IDentity). The file system writes each object to the physical storage space of the logical volume and records the storage location of each object. When a client requests data access, the file system can provide access based on the storage location information of each object. The storage system allocates physical storage space to logical volumes by pre-dividing the physical storage space into stripes based on the estimated capacity of the objects to be stored in the logical volume (this estimate often has a large margin relative to the actual capacity of the objects to be stored) and the Redundant Array of Independent Disks (RAID) groupings. A logical volume can be understood as a stripe, thereby allocating physical storage space to the logical volume.

[0077] A database, in short, can be thought of as a digital filing cabinet—a place where electronic files are stored, allowing users to add, query, update, and delete data. A database is a collection of data stored in a specific way, shared by multiple users, with minimal redundancy, and independent of applications.

[0078] See also Figure 1 , which is a schematic diagram of an implementation environment provided by an embodiment of the present invention. The implementation environment may include a client 110, a server 120 and a database 130.

[0079] The client 110 and the server 120 , as well as the server 120 and the database 130 , may be connected and communicated via a network.

[0080] The client 110 includes but is not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, in-vehicle clients, aircraft, etc. An application with human-computer interaction functions runs on the client 110, and the application can launch virtual item distribution activities for different business scenarios, such as flash sales, raffles, and reward activities for completing tasks. Optionally, the client 110 can obtain an image to be recognized; the image to be recognized includes text; perform visual recognition processing on the image to be recognized to obtain visual feature information of the image to be recognized; perform character position alignment processing on the visual feature information to obtain a character information sequence corresponding to the text; wherein the position of the character information of each word in the text in the character information sequence is the same as the position of each word in the text; and determine the semantic information of the text based on the character information of each word and the character information of the associated words of each word in the character information sequence.

[0081] The server 120 can perform semantic extraction of text for the image to be recognized provided by the client 110. Specifically, the server can obtain the image to be recognized; the image to be recognized includes text; perform visual recognition processing on the image to be recognized to obtain visual feature information of the image to be recognized; the visual feature information includes at least one of edge feature information, corner feature information, color feature information, texture feature information and shape feature information of the image to be recognized; perform character position alignment processing on the visual feature information to obtain a character information sequence corresponding to the text; wherein the position of the character information of each word in the text in the character information sequence is the same as the position of each word in the text; and determine the semantic information of the text based on the character information of each word and the character information of the associated words of each word in the character information sequence.

[0082] The database 130 may include an in-memory database and a relational database. It should be noted that the servers, databases, nodes, etc. of the embodiments of the present invention may be independent physical servers, or they may be server clusters or distributed systems composed of multiple physical servers. They may also be cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0083] In one exemplary embodiment, client 110, server 120, and database 130 may all be node devices in a blockchain system, capable of sharing acquired and generated information with other node devices in the blockchain system, thereby enabling information sharing among multiple node devices. Multiple node devices in a blockchain system may be configured with the same blockchain, which is composed of multiple blocks, with adjacent blocks having an associated relationship. This allows any tampering of data in any block to be detected by the next block, thereby preventing tampering of data in the blockchain and ensuring the security and reliability of the data in the blockchain.

[0084] See also Figure 2 , which is a flow chart of a semantic information recognition method provided by an embodiment of the present invention, which can be applied to Figure 1 In the implementation environment shown, the execution subject of this method can be Figure 1 The server that performs semantic information recognition can also perform semantic recognition on the client or other server nodes. It should be noted that this specification provides method operation steps as described in the embodiments or flow charts, but more or fewer operation steps may be included based on conventional or non-creative work. The order of steps listed in the embodiments is only one way of executing the steps among many steps, and does not represent the only execution order. When the system or product is actually executed, it can be executed sequentially or in parallel according to the method shown in the embodiments or the drawings (for example, in a parallel processor or multi-threaded processing environment). Specifically, Figure 2 As shown, the method may include:

[0085] S201, obtaining an image to be recognized; the image to be recognized includes text.

[0086] Since the present application is used to identify text in an image and determine the semantic information of the text, in an embodiment of the present application, the server can obtain an image to be identified containing text. Optionally, the characters in the text can be Chinese characters or other characters.

[0087] In an optional embodiment, the image to be recognized containing text may be a real image captured by a shooting terminal, or may be a synthetic image containing text synthesized by other terminals.

[0088] In an optional embodiment, the text in the image to be identified containing text can be located at any position in the image to be identified, and can also be evenly spread in the image to be identified, for example, the image to be identified is a screenshot of a page of a journal, and can also occupy the main part of the image to be identified.

[0089] Optionally, the images to be recognized can be of different sizes, i.e., some images to be recognized may have a resolution of 512*128, while others may have a resolution of 1024*128. The first digit in the resolution may represent the number of pixels in the image's width, and the second digit may represent the number of pixels in the image's height.

[0090] S203 , performing visual recognition processing on the image to be recognized to obtain visual feature information of the image to be recognized; the visual feature information includes at least one of edge feature information, corner feature information, color feature information, texture feature information, and shape feature information of the image to be recognized.

[0091] In an embodiment of the present application, the server can perform visual recognition processing on the image to be recognized to obtain visual feature information of the image to be recognized, wherein the visual feature information does not contain any text character information and is the output information of the image vision.

[0092] Optionally, the visual feature information includes at least one of edge feature information, corner feature information, color feature information, texture feature information, and shape feature information of the image to be identified. The edge feature information may include step edge feature information and line edge feature information. Step edge feature information may indicate that there is a significant difference in the grayscale values ​​of pixels on either side of a discontinuity in image brightness. Line edge feature information may indicate that the image brightness changes from one grayscale value to another and then quickly returns to the original grayscale value. Corner feature information may indicate points with high curvature in the image. Points with high curvature are typically composed of areas with large curvature or edge intersections. Color feature information is a feature obtained based on pixels and can indicate the surface properties of the object corresponding to the feature image or image region, and is not affected by changes such as image rotation and translation. Texture feature information is not based on a single pixel, but rather is derived from a region composed of multiple pixels and is used to indicate the surface properties of the object corresponding to the feature image or image region. The shape feature information may include contour feature information and region feature information of the object in the image, wherein the contour feature information mainly targets the outer boundary of the object in the image, and the region feature information mainly involves the shape region information of the entire image.

[0093] In an optional embodiment, the server may call an independent visual feature extractor to perform visual recognition processing on the image to be recognized to obtain visual feature information of the image to be recognized. The independent visual feature extractor may be constructed based on a recurrent neural network (RNN).

[0094] In another optional embodiment, the processes involved in steps S203-S207 are implemented using a trained target semantic recognition network. Therefore, after obtaining the image to be recognized, the server can call upon the trained target semantic recognition network to perform subsequent visual feature information recognition, character information sequence acquisition, and ultimately, determination of the semantic information of the text.

[0095] Figure 3 is a structural diagram of a trained target semantic recognition network provided by an embodiment of the present invention, such as Figure 3 As shown, the target semantic recognition network may include a visual feature recognition module, a position alignment module and a character semantic recognition module.

[0096] Optionally, the server may perform visual recognition processing on the image to be recognized based on the visual feature recognition module in the target semantic recognition network to obtain visual feature information of the image to be recognized.

[0097] In an embodiment of the present application, the visual feature recognition module may include several convolutional neural networks. Optionally, several neural convolutional networks may be connected in series to form the visual feature recognition module, or they may be connected in series and in parallel to form the visual feature recognition model. Each neural convolutional network may include a convolutional layer and a downsampling layer so that the input image to be recognized is converted from the original resolution (for example, 32 pixels high and 512 pixels wide) to visual feature information with a height of 1 and a width of 64, that is, one-dimensional visual feature information is output.

[0098] In an optional embodiment, each neural convolutional network may include not only convolutional layers and downsampling layers, but also mixing layers and maximum pooling layers. In order to avoid the problem of gradient vanishing and gradient exploding due to the depth of the deep learning network, data initialization (normlized initializatiton) and regularization (batch normlization) may be used to solve the gradient problem. However, as the network depth increases, another problem may arise, namely, the degradation of network performance. That is, as the network depth increases, the error rate increases. Therefore, it is possible to use Figure 4 The residual structure shown is used to solve the degradation problem. Figure 4This is a schematic diagram of a residual structure provided by an embodiment of the present invention. It not only solves the gradient problem but also improves network performance. Furthermore, in practical applications, data is often not linearly separable. To introduce nonlinear factors, activation layers can be added to convolutional neural networks.

[0099] Based on the above explanation, the visual feature information does not contain any text character information, but is the output information of the image vision. It can be seen that the neural convolution network of the visual feature recognition module does not know the position of each word in the text in the image to be recognized, but is used to extract the visual feature information of the image. In other words, the output visual feature information has nothing to do with each word in the text. However, due to the natural local correlation of convolution, such as the convolution kernel of 3*3, it processes 9 pixels at a time. Therefore, the features of each single word in the text should be distributed near the corresponding position. For example, the features of each single word in "Today is sunny" (such as the features of the word "today") should be distributed near the position corresponding to the word "today" and will not shift too far. For example, the features of the word "today" can also be distributed in a small part at the position corresponding to the word "sun", but it is basically impossible to be distributed at the position corresponding to the word "day". In this way, the foundation can be laid for the position alignment of subsequent single words.

[0100] S205 , performing character position alignment processing on the visual feature information to obtain a character information sequence corresponding to the text; wherein the position of the character information of each word in the text in the character information sequence is the same as the position of each word in the text.

[0101] In an optional embodiment, since the visual feature information can be one-dimensional visual feature information with a height of 1 and a width of 64, it can be understood that the visual feature information includes 64 sub-visual feature information. Since the features of each word in the text should be distributed near the corresponding position, the server can calculate the feature similarity between each sub-visual feature information in the 64 sub-visual feature information and the nearby sub-visual feature information, and perform character position alignment processing on the visual feature information based on the feature similarity to obtain the character information sequence corresponding to the text.

[0102] In another optional embodiment, the server may perform character position alignment processing on the visual feature information based on the position alignment module in the target semantic recognition network to obtain a character information sequence corresponding to the text.

[0103] Among them, the position of the character information of each word in the text in the character information sequence is the same as the position of each word in the text. Taking the above "Today is a sunny day" as an example, the character information sequence is the character information of "today", the character information of "day", the character information of "is", the character information of "sunny" and the character information of "sky", which is the same as the position of each word in the text.

[0104] In an optional embodiment, the server may perform character position alignment processing through the classification capability of the position alignment module to obtain a character information sequence corresponding to the text. Figure 5 FIG. 1 is a flow chart of determining a character information sequence corresponding to a text according to an embodiment of the present invention. Figure 5 Shown, including:

[0105] In S501, based on the position alignment module in the target semantic recognition network, the sub-visual feature information corresponding to each word in the text included in the image to be recognized in the visual feature information of the image to be recognized is classified to obtain the character information of each word in the text.

[0106] In an embodiment of the present application, the position alignment module in the target semantic recognition network is constructed based on the Transformer autocorrelation framework, which includes a query component, a keyword component and a value component.

[0107] In an embodiment of the present application, in order to apply the Transformer autocorrelation framework to the position alignment module, and taking into account the number of characters in the text, the amount of output information of the query component can be fixed to be the same as the width of the visual feature information output by the previous layer, that is, the visual feature recognition module, such as 64. In this way, the output information of the query component is at most 64.

[0108] Since the target semantic recognition network has been trained, the server can classify the sub-visual feature information corresponding to each word in the text included in the image to be recognized in the visual feature information of the image to be recognized based on the module parameters of the position alignment module in the target semantic recognition network to obtain the character information of each word in the text.

[0109] Specifically, in the visual feature information of the image to be identified, the sub-visual feature information corresponding to the first word in the text included in the image to be identified is classified to obtain the character information of the first word in the text, the sub-visual feature information corresponding to the second word in the text included in the image to be identified is classified to obtain the character information of the second word in the text, the sub-visual feature information corresponding to the third word in the text included in the image to be identified is classified to obtain the character information of the third word in the text... and the sub-visual feature information corresponding to the last word in the text included in the image to be identified is classified to obtain the character information of the last word in the text.

[0110] That is to say, after training is completed, the position alignment module under the Transformer autocorrelation framework can classify the output features of the first query vector as the first character information in the image to be recognized, the output features of the second query vector as the second character information in the image to be recognized, and so on.

[0111] In S503 , the position identifier of the character information of each word in the text is determined based on the position of the sub-visual feature information corresponding to each word in the text in the visual feature information of the image to be recognized.

[0112] As mentioned above, the server can determine the sub-visual feature information corresponding to each word in the text included in the image to be recognized based on the module parameters of the position alignment module in the target semantic recognition network. Therefore, the server can determine the position identification of the character information of each word in the text based on the position of the sub-visual feature information corresponding to each word in the text in the visual feature information of the image to be recognized.

[0113] For example, the server can determine the position identifier of the character information whose most sub-visual feature information is located at the first single word position and nearby positions as the first position, determine the position identifier of the character information whose most sub-visual feature information is located at the second single word position and nearby positions as the second position... and determine the position identifier of the character information whose most sub-visual feature information is located at the last single word position and nearby positions as the last position.

[0114] The information contained in the first position, the second position... and the last position may be digital information, such as 1, 2...N, where 1 indicates that the corresponding character information is in the first position, 2 indicates that the corresponding character information is in the second position, and so on.

[0115] Alternatively, the information contained in the first position, the second position, ... and the last position may be an identifier of the previous character information and / or an identifier of the next character information.

[0116] In S505 , a character information sequence corresponding to the text contained in the image to be recognized is determined based on the character information of each word in the text and the position identifier of the character information of each word in the text.

[0117] In an embodiment of the present application, the server can determine the position of the character information of each word in the character information sequence corresponding to the final text based on the position identifier of the character information of each word in the text, and arrange the character information of each word according to the position to obtain the character information sequence corresponding to the text contained in the image to be identified.

[0118] In another optional embodiment, the server may perform character position alignment processing using a character classifier included in the position alignment module to obtain a character information sequence corresponding to the text. Specifically, the server may use the character classifier to classify the visual feature information, grouping sub-visual feature information belonging to the same character together and determining it as the character information corresponding to the character, thereby achieving the purpose of single-character position alignment.

[0119] S207: Determine the semantic information of the text based on the character information of each word and the character information of the associated words of each word in the character information sequence.

[0120] In an optional embodiment, the server can determine the semantic information of the text corresponding to the character information sequence corresponding to the text contained in the image to be recognized based on a character information sequence and text semantic information mapping table. Since the semantic information of the text is determined, the server can obtain the text.

[0121] However, the character information sequence and text semantic information mapping table cannot include all possible character information sequences and text semantic information, especially as the diversity and complexity of text applications increase.

[0122] Based on this, in another optional embodiment, the server can perform semantic recognition on the character information of each word and the character information of the associated words of each word in the character information sequence based on the character semantic recognition module in the trained target semantic recognition network to obtain the semantic information of the text.

[0123] Optionally, the character semantic recognition module in the target semantic recognition network can be built based on the encoder in the Transformer autocorrelation framework, predicting the semantics of the current character based on the contextual semantics of the current character, thereby obtaining the semantic information of the entire text. The encoder in the Transformer autocorrelation framework includes a 12-layer multi-head self-attention layer, which is a 12-head self-attention layer.

[0124] Since the character semantic recognition module is constructed based on the encoder in the Transformer autocorrelation framework with a self-attention layer, and in the subsequent training process, a mask module is used to cover the character information of the character information sequence, and then the character semantic recognition module is trained to guess the semantic information of the middle covered text based on the character information of the previous and next texts, even if the text in the image to be recognized has an unclear text model or typos, the present application can obtain accurate text and text semantic information. That is, the solution of the present application also has specific text error correction capabilities.

[0125] In summary, the present application can realize text recognition and semantic acquisition of images to be recognized containing text through a trained single network - the target semantic recognition network. It does not need to use the existing technology to combine the character information of each word and the character information of the associated words of each word through two independent networks in the optical character recognition model and the semantic extraction model, resulting in multi-model independent training, and then conduct joint debugging and deployment, which increases the training and deployment costs.

[0126] The embodiment of the present application also includes a method for training a target semantic recognition network. Figure 6 : is a flow chart of a training target semantic recognition network provided by an embodiment of the present invention, such as Figure 6 Shown, including:

[0127] S601, obtaining a sample image set; the sample image set includes multiple sample images and preset semantic information corresponding to each sample image; the multiple sample images have the same size; each sample image in the multiple sample images includes sample text.

[0128] In an embodiment of the present application, since the present application is used to recognize text in an image and determine the semantic information of the text, in an embodiment of the present application, during the training phase, the multiple sample images obtained by the server may be images containing sample text. Optionally, the characters in the sample text may be Chinese characters or other characters.

[0129] In an optional embodiment, the sample image containing the sample text may be a real image captured by a shooting terminal, or may be a synthetic image containing the sample text synthesized by other terminals.

[0130] In an optional embodiment, the sample text in the sample image containing the sample text can be located at any position in the sample image, and can also be evenly spread in the sample image, for example, the sample image is a screenshot of a page of a journal, and can also occupy the main part of the image to be identified.

[0131] In an optional embodiment, the original semantics are obtained by network training. The target semantic recognition network is a supervised training. Therefore, each sample image in the sample image set can carry preset semantic information.

[0132] In an optional embodiment, the sizes of all sample images in the sample image set can be unlimited, for example, some are 512*32 and some are 1024*128. However, since training involves combining multiple images together, in order to increase training speed, it is possible to ensure that all sample images in the sample image set have the same size, for example, 512*312.

[0133] In an optional embodiment of determining the size of the sample image, the server may obtain a plurality of original images, wherein each of the plurality of original images includes sample text.

[0134] During the training process, in order to avoid introducing too much feature information irrelevant to the text, the server can crop the text in the original image. Specifically, the server can perform text position detection on multiple original images to obtain the position information of the sample text in each original image, and crop each original image based on the position information of the sample text in each original image to obtain a cropped image containing the sample text. Subsequently, the server can perform image size conversion on each cropped image to obtain a sample image set. Each sample image in the sample image set is an image of the same size after image size conversion.

[0135] Optionally, image size conversion processing includes operations such as splicing, pixel padding, and scaling.

[0136] S603 , performing visual recognition processing on the sample image set through the visual feature recognition module in the original semantic recognition network to obtain visual feature information of each sample image, wherein the visual feature information of each sample image is one-dimensional information.

[0137] In an embodiment of the present application, the server can perform visual recognition processing on the sample image set through the visual feature recognition module in the original semantic recognition network to obtain one-dimensional visual feature information of each sample image, which facilitates subsequent character alignment operations.

[0138] Optionally, the visual feature information of each sample image may also be multi-dimensional. Compared with one-dimensional visual feature information, the server needs to use more resources to implement subsequent character alignment operations.

[0139] S605 , performing character position alignment processing on the visual feature information of each sample image based on the position alignment module in the original semantic recognition network to obtain a character information sequence corresponding to the sample text contained in each sample image.

[0140] Refer to the above Figure 5 A flow chart of the character information sequence corresponding to the text involved. In an embodiment of the present application, the server can classify the sub-visual feature information corresponding to each word in the sample text included in each sample image in the visual feature information of each sample image based on the position alignment module in the original semantic recognition network to obtain the character information of each word.

[0141] Specifically, in the visual feature information of the sample image, the sub-visual feature information corresponding to the first character in the sample text included in the sample image is classified to obtain the character information of the first character in the sample text, the sub-visual feature information corresponding to the second character in the sample text included in the sample image is classified to obtain the character information of the second character in the sample text, the sub-visual feature information corresponding to the third character in the sample text included in the sample image is classified to obtain the character information of the third character in the sample text... The sub-visual feature information corresponding to the last character in the sample text included in the sample image is classified to obtain the character information of the last character in the sample text.

[0142] Subsequently, the server can determine the position identifier of the character information of each character based on the position of the sub-visual feature information corresponding to each character in the visual feature information of each sample image, and determine the character information sequence corresponding to the sample text included in each sample image based on the character information of each character and the position identifier of the character information of each character.

[0143] Among them, the position of the character information of each character in the sample text in the character information sequence corresponding to the sample text is the same as the position of each character in the sample text in the sample text. For example, assuming the sample text is "Exercise guides a healthy life", the character information sequence corresponding to the sample text is successively the character information of "运", the character information of "动", the character information of "引", the character information of "导", the character information of "健", the character information of "康", the character information of "生", and the character information of "活", which is the same as the position of each character in the sample text.

[0144] In another optional embodiment, during the training process, a mask generation module plug-in can be set before the position alignment module, and after the training is completed, the plug-in is removed. One end of the mask generation module plug-in is connected to the visual feature recognition module to obtain visual feature information. The mask generation module is used to generate a mask and the inverse code of the mask. One branch of the mask generation module covers the mask on the visual feature information and then sends it to the character classifier in the position alignment module to classify non-covered characters. Another branch covers the inverse code of the mask on the visual feature information and sends it to the character classifier in the position alignment module to classify non-covered characters. In this way, as the training progresses, the character classifier in the position alignment module will gradually align the generated mask to the single-character position, so as to achieve the purpose of single-character alignment.

[0145] S607, perform character information masking processing on the character information sequence corresponding to each sample image based on the mask module in the original semantic recognition network to obtain the masked character information sequence corresponding to each sample image.

[0146] In the embodiments of the present application, in order to enable the trained target semantic recognition network to not only recognize text but also determine the semantics of the text, a plug-in, i.e., a masking module, can be set between the position alignment module and the character semantic recognition module during the training process. This masking module is used to mask the character information in the character information sequence corresponding to the sample text, facilitating the subsequent character semantic recognition module to predict the semantics of the masked character information based on the character information before the masked character information.

[0147] Based on this, the server can perform character information masking processing on the character information sequence corresponding to each sample image based on the masking module in the original semantic recognition network, obtaining a masked character information sequence corresponding to each sample image.

[0148] In order to improve the model performance of the finally trained target semantic recognition network, increase the diversity of the text, and facilitate masking, in the embodiments of the present application, the server can add interval character information after the character information sequence corresponding to each sample image to obtain a first information sequence corresponding to each sample image. Taking the sample text "Exercise guides a healthy life" as an example, the first information sequence corresponding to the sample text successively includes the character information of "运", the character information of "动", the character information of "引", the character information of "导", the character information of "健", the character information of "康", the character information of "生", the character information of "活", and the interval character information.

[0149] Subsequently, the server can splice the N first information sequences corresponding to the N sample images in the sample image set to obtain a second information sequence, where N is a positive integer greater than 1. Optionally, if there is a limit to the length of the second information sequence, such as 512 characters, the range of N can be determined, and then, based on the range of N, several first information sequences can be determined for splicing to obtain the second information sequence.

[0150] Next, the server can perform character information masking processing on the first information sequence and the second information sequence respectively based on the masking module in the original semantic recognition network to obtain a set of masked character information sequences corresponding to the sample image set. Among them, the set of masked character information sequences includes the first information sequence masked by the mask, including the second information sequence masked by the mask.

[0151] Among them, the first information sequence masked by the mask included in the set of masked character information sequences can be the first information sequence masked by the mask corresponding to each sample image, or can be obtained by masking the remaining first information sequences after removing the second information sequence corresponding to the second information sequence.

[0152] In this way, the present application can splice sentences of different lengths together to form a set of masked character information sequences combining long sentences and short sentences.

[0153] In an embodiment of the present application, in order to provide robustness and stability of the target semantic recognition model and enhance the prediction accuracy of texts with consecutive missing or wrong characters, the server can perform single-word character information masking processing on the first information sequence based on the mask module in the original semantic recognition network to obtain a first mask information sequence; and perform single-word character information masking and continuous multi-word character information masking processing on the second information sequence based on the mask module in the original semantic recognition network to obtain a second mask information sequence. Alternatively, the server can perform single-word character information masking and continuous multi-word character information masking processing on the first information sequence based on the mask module in the original semantic recognition network to obtain a first mask information sequence; and perform single-word character information masking and continuous multi-word character information masking processing on the second information sequence based on the mask module in the original semantic recognition network to obtain a second mask information sequence. Subsequently, the mask character information sequence set corresponding to the sample image set is determined based on the first mask information sequence and the second mask information sequence.

[0154] In order to prevent subsequent accurate prediction from being affected by too many missing characters, the proportion of mask characters in the first mask information sequence and the second mask information sequence is less than or equal to a preset value.

[0155] S609 , performing semantic recognition on the mask character information sequence corresponding to each sample image based on the character semantic recognition module in the original semantic recognition network to obtain predicted semantic information corresponding to each sample image.

[0156] The server can perform semantic recognition on the mask character information sequence corresponding to each sample image based on the character semantic recognition module in the original semantic recognition network to obtain predicted semantic information corresponding to each sample image.

[0157] Since there is masked character information in the masked character information sequence, the character semantic recognition module in the original semantic recognition network guesses the semantic information of the masked text in the middle based on the character information of the previous and next texts during the training process, and obtains the predicted semantic information corresponding to each sample image.

[0158] S611 , training an original semantic recognition network based on the predicted semantic information corresponding to each sample image and the preset semantic information corresponding to each sample image to obtain a target semantic recognition network.

[0159] In an embodiment of the present application, the server can determine a semantic loss function based on the predicted semantic information corresponding to each sample image and the preset semantic information corresponding to each sample image, and update the network parameters of the original semantic recognition network based on the semantic loss function to obtain an updated original semantic recognition network. The original semantic recognition network is trained in a loop until an iteration termination condition is met, and the target semantic recognition network is obtained by combining the visual feature recognition module, position alignment module, and character semantic recognition module in the trained original semantic recognition network.

[0160] Among them, when the number of cyclic training reaches the preset number, or when the semantic loss function is less than the preset loss function, and the iteration termination condition is met, the server can obtain the target semantic recognition network by combining the visual feature recognition module, position alignment module and character semantic recognition module in the trained original semantic recognition network, that is, unplug the mask module plug-in to obtain the trained target semantic recognition network.

[0161] See also Figure 7 , which is a schematic diagram of the structure of a semantic information recognition device provided by an embodiment of the present invention, the device has the function of implementing the semantic information recognition method in the above method embodiment, and the function can be implemented by hardware or by hardware executing corresponding software. Figure 7 As shown, the semantic information recognition device 700 may include:

[0162] The image acquisition module 701 is used to acquire an image to be recognized; the image to be recognized includes text;

[0163] A visual processing module 702 is configured to perform visual recognition processing on the image to be recognized to obtain visual feature information of the image to be recognized; the visual feature information includes at least one of edge feature information, corner feature information, color feature information, texture feature information, and shape feature information of the image to be recognized;

[0164] An alignment processing module 703 is configured to perform character position alignment processing on the visual feature information to obtain a character information sequence corresponding to the text; wherein the position of the character information of each word in the text in the character information sequence is the same as the position of each word in the text;

[0165] The semantic determination module 704 is configured to determine the semantic information of the text based on the character information of each word and the character information of the associated words of each word in the character information sequence.

[0166] In some possible embodiments,

[0167] A visual processing module is used to perform visual recognition processing on the image to be recognized based on the visual feature recognition module in the target semantic recognition network to obtain visual feature information of the image to be recognized;

[0168] An alignment processing module is used to perform character position alignment processing on the visual feature information based on the position alignment module in the target semantic recognition network to obtain a character information sequence corresponding to the text;

[0169] The semantic determination module is used to perform semantic recognition on the character information of each word and the character information of the associated words of each word in the character information sequence based on the character semantic recognition module in the target semantic recognition network to obtain the semantic information of the text.

[0170] In some possible embodiments, the method further includes a network training module for:

[0171] Acquire a sample image set; the sample image set includes a plurality of sample images and preset semantic information corresponding to each sample image; the plurality of sample images have the same size; each of the plurality of sample images includes sample text;

[0172] The visual feature recognition module in the original semantic recognition network performs visual recognition processing on the sample image set to obtain the visual feature information of each sample image; the visual feature information of each sample image is one-dimensional information;

[0173] Based on the position alignment module in the original semantic recognition network, the visual feature information of each sample image is aligned with the character position to obtain the character information sequence corresponding to the sample text contained in each sample image;

[0174] Based on the mask module in the original semantic recognition network, the character information sequence corresponding to each sample image is masked to obtain the masked character information sequence corresponding to each sample image;

[0175] Based on the character semantic recognition module in the original semantic recognition network, semantic recognition is performed on the mask character information sequence corresponding to each sample image to obtain the predicted semantic information corresponding to each sample image;

[0176] The original semantic recognition network is trained based on the predicted semantic information corresponding to each sample image and the preset semantic information corresponding to each sample image to obtain the target semantic recognition network.

[0177] In some possible embodiments, the network training module is used to:

[0178] Based on the position alignment module in the original semantic recognition network, the sub-visual feature information corresponding to each word in the sample text included in each sample image is classified into character information of each word;

[0179] Determining a position identifier of the character information of each word based on the position of the sub-visual feature information corresponding to each word in the visual feature information of each sample image;

[0180] Determining a character information sequence corresponding to a sample text contained in each sample image based on the character information of each word and a position identifier of the character information of each word;

[0181] The position of the character information of each word in the sample text in the corresponding character information sequence of the sample text is the same as the position of each word in the sample text.

[0182] In some possible embodiments, the network training module is used to:

[0183] Adding spacing character information after the character information sequence corresponding to each sample image to obtain a first information sequence corresponding to each sample image;

[0184] In the sample image set, N first information sequences corresponding to N sample images are spliced ​​together to obtain a second information sequence, where N is a positive integer greater than 1;

[0185] Based on the mask module in the original semantic recognition network, character information masking processing is performed on the first information sequence and the second information sequence respectively to obtain a mask character information sequence set corresponding to the sample image set.

[0186] In some possible embodiments, the network training module is used to:

[0187] Performing single-word character information masking processing on the first information sequence based on the mask module in the original semantic recognition network to obtain a first masked information sequence;

[0188] performing single-word character information masking and continuous multi-word character information masking processing on the second information sequence based on the masking module in the original semantic recognition network to obtain a second masked information sequence;

[0189] Determining a mask character information sequence set corresponding to the sample image set based on the first mask information sequence and the second mask information sequence;

[0190] The proportion of mask characters in the first mask information sequence and the second mask information sequence is less than or equal to a preset value.

[0191] In some possible embodiments, the network training module is used to:

[0192] Determine a semantic loss function based on the predicted semantic information corresponding to each sample image and the preset semantic information corresponding to each sample image;

[0193] Based on the semantic loss function, the network parameters of the original semantic recognition network are updated to obtain an updated original semantic recognition network;

[0194] The original semantic recognition network is trained cyclically until the iteration termination condition is met;

[0195] The target semantic recognition network is obtained by combining the visual feature recognition module, position alignment module and character semantic recognition module in the trained original semantic recognition network.

[0196] In some possible embodiments, the network training module is used to:

[0197] Acquire a plurality of original images; each of the plurality of original images includes sample text;

[0198] Performing text position detection on multiple original images to obtain position information of sample text in each original image;

[0199] Cropping each original image based on position information of the sample text in each original image to obtain a cropped image containing the sample text;

[0200] An image size conversion process is performed on each cropped image to obtain a sample image set; each sample image in the sample image set is an image with the same size after the image size conversion.

[0201] It should be noted that the apparatus provided in the above embodiments, when implementing its functions, is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0202] An embodiment of the present invention provides an electronic device, which includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the semantic information recognition method provided in the above method embodiment.

[0203] The memory can be used to store software programs and modules. The processor executes various functional applications and semantic information recognition by running the software programs and modules stored in the memory. The memory can mainly include a program storage area and a data storage area. The program storage area can store the operating system, application programs required for the functions, etc.; the data storage area can store data created based on the use of the device, etc. In addition, the memory can include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory can also include a memory controller to provide the processor with access to the memory.

[0204] The method embodiments provided in the embodiments of the present invention can be executed in a computer terminal, a server or a similar computing device. Taking running on a server as an example, Figure 8 This is a hardware structure diagram of a server running a semantic information recognition method provided by an embodiment of the present invention, such as Figure 8 As shown, the server 1000 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPUs) 1010 (the processor 1010 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 1030 for storing data, and one or more storage media 1020 (such as one or more mass storage devices) for storing application programs 1023 or data 1022. Among them, the memory 1030 and the storage medium 1020 can be temporary storage or permanent storage. The program stored in the storage medium 1020 may include one or more modules, each module may include a series of instruction operations on the server. Furthermore, the central processing unit 1010 can be configured to communicate with the storage medium 1020 to execute a series of instruction operations in the storage medium 1020 on the server 1000. The server 1000 may also include one or more power supplies 1060, one or more wired or wireless network interfaces 1050, one or more input and output interfaces 1040, and / or one or more operating systems 1021, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0205] The input / output interface 1040 can be used to receive or send data via a network. Specific examples of the aforementioned network may include a wireless network provided by the communication provider of the server 1000. In one embodiment, the input / output interface 1040 includes a network interface controller (NIC), which can be connected to other network devices via a base station to communicate with the Internet. In one embodiment, the input / output interface 1040 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0206] It can be understood by those skilled in the art that Figure 8 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 8 More or fewer components than shown, or with Figure 8 Different configurations shown.

[0207] An embodiment of the present invention also provides a computer-readable storage medium, which can be set in an electronic device to store at least one instruction or at least one program related to implementing a semantic information recognition method. The at least one instruction or the at least one program is loaded and executed by the processor to implement the semantic information recognition method provided by the above method embodiment.

[0208] Optionally, in this embodiment, the above-mentioned storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store program codes.

[0209] It should be noted that the order in which the embodiments of the present invention are described above is for illustrative purposes only and does not represent the superiority or inferiority of the embodiments. The above description is of specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0210] An embodiment of the present invention further provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the above-described semantic information recognition method.

[0211] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences from other embodiments. In particular, the device embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

[0212] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0213] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A semantic information recognition method, characterized in that: The method comprises: Acquire an image to be recognized; the image to be recognized includes text; performing visual recognition processing on the image to be recognized based on a visual feature recognition module in a target semantic recognition network to obtain visual feature information of the image to be recognized; the visual feature information includes at least one of edge feature information, corner feature information, color feature information, texture feature information, and shape feature information of the image to be recognized; performing character position alignment processing on the visual feature information based on a position alignment module in the target semantic recognition network to obtain a character information sequence corresponding to the text; wherein the position of the character information of each word in the text in the character information sequence is the same as the position of each word in the text; Based on the character semantic recognition module in the target semantic recognition network, semantic recognition is performed on the character information of each character and the character information of the associated characters of each character in the character information sequence to obtain the semantic information of the text; the target semantic recognition network is obtained by training the original semantic recognition network based on the predicted semantic information corresponding to each sample image in the sample image set and the preset semantic information corresponding to each sample image; the predicted semantic information is obtained by semantically recognizing the mask character information sequence corresponding to each sample image based on the character semantic recognition module in the original semantic recognition network; the mask character information sequence corresponding to each sample image is obtained in the following manner: adding interval character information after the character information sequence corresponding to each sample image to obtain the first information sequence corresponding to each sample image; splicing the N first information sequences corresponding to the N sample images in the sample image set to obtain the second information sequence; N is a positive integer greater than 1; based on the mask module in the original semantic recognition network, character information masking processing is performed on the first information sequence and the second information sequence respectively to obtain a set of mask character information sequences corresponding to the sample image set.

2. The semantic information recognition method according to claim 1, characterized in that: The method further comprises: Acquire the sample image set; the sample image set includes a plurality of sample images and preset semantic information corresponding to each sample image; the plurality of sample images have the same size; each of the plurality of sample images includes sample text; Performing visual recognition processing on the sample image set through the visual feature recognition module in the original semantic recognition network to obtain visual feature information of each sample image; the visual feature information of each sample image is one-dimensional information; Performing character position alignment processing on the visual feature information of each sample image based on the position alignment module in the original semantic recognition network to obtain a character information sequence corresponding to the sample text contained in each sample image; performing character information masking processing on the character information sequence corresponding to each sample image based on the mask module in the original semantic recognition network to obtain a masked character information sequence corresponding to each sample image; Performing semantic recognition on the mask character information sequence corresponding to each sample image based on the character semantic recognition module in the original semantic recognition network to obtain predicted semantic information corresponding to each sample image; The original semantic recognition network is trained based on the predicted semantic information corresponding to each sample image and the preset semantic information corresponding to each sample image to obtain the target semantic recognition network.

3. The semantic information recognition method according to claim 2, characterized in that: The position alignment module in the original semantic recognition network performs character position alignment processing on the visual feature information of each sample image to obtain a character information sequence corresponding to the sample text contained in each sample image, including: Based on the position alignment module in the original semantic recognition network, the sub-visual feature information corresponding to each word in the sample text included in each sample image is classified in the visual feature information of each sample image to obtain character information of each word; Determining a position identifier of the character information of each word based on the position of the sub-visual feature information corresponding to each word in the visual feature information of each sample image; Determining a character information sequence corresponding to the sample text contained in each sample image based on the character information of each word and a position identifier of the character information of each word; The position of the character information of each word in the sample text in the corresponding character information sequence of the sample text is the same as the position of each word in the sample text in the sample text.

4. The semantic information recognition method according to claim 1, characterized in that: The masking module in the original semantic recognition network performs character information masking processing on the first information sequence and the second information sequence respectively to obtain a masked character information sequence set corresponding to the sample image set, including: performing single-word character information masking processing on the first information sequence based on the masking module in the original semantic recognition network to obtain a first masked information sequence; performing single-word character information masking and continuous multi-word character information masking processing on the second information sequence based on the masking module in the original semantic recognition network to obtain a second masked information sequence; Determining a mask character information sequence set corresponding to the sample image set based on the first mask information sequence and the second mask information sequence; The proportion of mask characters in the first mask information sequence and the second mask information sequence is less than or equal to a preset value.

5. The semantic information recognition method according to claim 2, characterized in that: The step of training the original semantic recognition network based on the predicted semantic information corresponding to each sample image and the preset semantic information corresponding to each sample image to obtain the target semantic recognition network includes: Determining a semantic loss function based on the predicted semantic information corresponding to each sample image and the preset semantic information corresponding to each sample image; updating the network parameters of the original semantic recognition network based on the semantic loss function to obtain an updated original semantic recognition network; Circularly training the original semantic recognition network until an iteration termination condition is met; The target semantic recognition network is obtained by combining the visual feature recognition module, the position alignment module and the character semantic recognition module in the trained original semantic recognition network.

6. The semantic information recognition method according to any one of claims 2 to 5, characterized in that: The acquiring of the sample image set comprises: Acquire a plurality of original images; each of the plurality of original images includes the sample text; Performing text position detection on the multiple original images to obtain position information of the sample text in each original image; Cropping each original image based on position information of the sample text in each original image to obtain a cropped image containing the sample text; An image size conversion process is performed on each cropped image to obtain the sample image set; each sample image in the sample image set is an image of the same size after the image size conversion.

7. A semantic information recognition device, characterized in that: The device comprises: An image acquisition module, configured to acquire an image to be recognized, wherein the image to be recognized includes text; a visual processing module, configured to perform visual recognition processing on the image to be recognized based on the visual feature recognition module in the target semantic recognition network to obtain visual feature information of the image to be recognized; the visual feature information includes at least one of edge feature information, corner feature information, color feature information, texture feature information, and shape feature information of the image to be recognized; an alignment processing module, configured to perform character position alignment processing on the visual feature information based on the position alignment module in the target semantic recognition network to obtain a character information sequence corresponding to the text; wherein the position of the character information of each word in the text in the character information sequence is the same as the position of each word in the text; A semantic determination module is used to perform semantic recognition on the character information of each character and the character information of the associated characters of each character in the character information sequence based on the character semantic recognition module in the target semantic recognition network to obtain the semantic information of the text; the target semantic recognition network is obtained by training the original semantic recognition network based on the predicted semantic information corresponding to each sample image in the sample image set and the preset semantic information corresponding to each sample image, and the predicted semantic information is obtained by performing semantic recognition on the mask character information sequence corresponding to each sample image based on the character semantic recognition module in the original semantic recognition network, and the mask character information sequence corresponding to each sample image is obtained in the following manner: adding interval character information after the character information sequence corresponding to each sample image to obtain the first information sequence corresponding to each sample image; splicing the N first information sequences corresponding to the N sample images in the sample image set to obtain the second information sequence; N is a positive integer greater than 1; performing character information masking processing on the first information sequence and the second information sequence respectively based on the mask module in the original semantic recognition network to obtain a set of mask character information sequences corresponding to the sample image set.

8. The semantic information recognition device according to claim 7, characterized in that: The device also includes a network training module, which is used to: Acquire the sample image set; the sample image set includes a plurality of sample images and preset semantic information corresponding to each sample image; the plurality of sample images have the same size; each of the plurality of sample images includes sample text; Performing visual recognition processing on the sample image set through the visual feature recognition module in the original semantic recognition network to obtain visual feature information of each sample image; the visual feature information of each sample image is one-dimensional information; Performing character position alignment processing on the visual feature information of each sample image based on the position alignment module in the original semantic recognition network to obtain a character information sequence corresponding to the sample text contained in each sample image; performing character information masking processing on the character information sequence corresponding to each sample image based on the mask module in the original semantic recognition network to obtain a masked character information sequence corresponding to each sample image; Performing semantic recognition on the mask character information sequence corresponding to each sample image based on the character semantic recognition module in the original semantic recognition network to obtain predicted semantic information corresponding to each sample image; The original semantic recognition network is trained based on the predicted semantic information corresponding to each sample image and the preset semantic information corresponding to each sample image to obtain the target semantic recognition network.

9. The semantic information recognition device according to claim 8, characterized in that: The position alignment module in the original semantic recognition network performs character position alignment processing on the visual feature information of each sample image to obtain a character information sequence corresponding to the sample text contained in each sample image, including: Based on the position alignment module in the original semantic recognition network, the sub-visual feature information corresponding to each word in the sample text included in each sample image is classified in the visual feature information of each sample image to obtain character information of each word; Determining a position identifier of the character information of each word based on the position of the sub-visual feature information corresponding to each word in the visual feature information of each sample image; Determining a character information sequence corresponding to the sample text contained in each sample image based on the character information of each word and a position identifier of the character information of each word; The position of the character information of each word in the sample text in the corresponding character information sequence of the sample text is the same as the position of each word in the sample text in the sample text.

10. The semantic information recognition device according to claim 7, characterized in that: The masking module in the original semantic recognition network performs character information masking processing on the first information sequence and the second information sequence respectively to obtain a masked character information sequence set corresponding to the sample image set, including: performing single-word character information masking processing on the first information sequence based on the masking module in the original semantic recognition network to obtain a first masked information sequence; performing single-word character information masking and continuous multi-word character information masking processing on the second information sequence based on the masking module in the original semantic recognition network to obtain a second masked information sequence; Determining a mask character information sequence set corresponding to the sample image set based on the first mask information sequence and the second mask information sequence; The proportion of mask characters in the first mask information sequence and the second mask information sequence is less than or equal to a preset value.

11. The semantic information recognition device according to claim 8, characterized in that: The step of training the original semantic recognition network based on the predicted semantic information corresponding to each sample image and the preset semantic information corresponding to each sample image to obtain the target semantic recognition network includes: Determining a semantic loss function based on the predicted semantic information corresponding to each sample image and the preset semantic information corresponding to each sample image; updating the network parameters of the original semantic recognition network based on the semantic loss function to obtain an updated original semantic recognition network; Circularly training the original semantic recognition network until an iteration termination condition is met; The target semantic recognition network is obtained by combining the visual feature recognition module, the position alignment module and the character semantic recognition module in the trained original semantic recognition network.

12. The semantic information recognition device according to any one of claims 8 to 11, characterized in that: The acquiring of the sample image set comprises: Acquire a plurality of original images; each of the plurality of original images includes the sample text; Performing text position detection on the multiple original images to obtain position information of the sample text in each original image; Cropping each original image based on position information of the sample text in each original image to obtain a cropped image containing the sample text; An image size conversion process is performed on each cropped image to obtain the sample image set; each sample image in the sample image set is an image of the same size after the image size conversion.

13. An electronic device, characterized in that: It includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the semantic information recognition method according to any one of claims 1 to 6.

14. A computer-readable storage medium, characterized in that The computer-readable storage medium stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the semantic information recognition method according to any one of claims 1 to 6.

15. A computer program product, characterized in that The computer program product includes at least one instruction or at least one program segment, and the at least one instruction or the at least one program segment is loaded and executed by a processor to implement the semantic information recognition method according to any one of claims 1 to 6.

Images