Text position detection method and system in image, medium and electronic device

Abstract

The application provides a text position detection method and system in an image, a medium and an electronic device. The method comprises: acquiring a to-be-detected image and a target detection character, and acquiring image features based on the to-be-detected image; acquiring time sequence visual features of the to-be-detected image based on the to-be-detected image and the image features; acquiring text enhancement features based on the image features and the target detection character; obtaining a segmentation map of the to-be-detected image based on the time sequence visual features and the text enhancement features; and positioning the position of the target detection character in the to-be-detected image according to the segmentation map. The technical scheme of the application uses text characters and time sequence semantics to detect the position of text, thereby improving the positioning accuracy of the position of text.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method and system for detecting the position of text in an image, as well as a medium and electronic device. Background Technology

[0002] Text location detection is a crucial step in text recognition, aiming to determine the location of text regions in an image. However, due to differences in font, size, color, and style of text, as well as complex background interference in images, detection techniques are complex, computationally resource-intensive, and sensitive to image quality. These issues affect the accuracy and efficiency of text detection, ultimately impacting the text recognition results.

[0003] Application content

[0004] The purpose of this application is to provide a method, system, medium, and electronic device for detecting the position of text in an image, which solves the technical problem of low accuracy and efficiency of text detection caused by the font, size, color, and complex background of the text in the image.

[0005] In a first aspect, this application provides a method for detecting the position of text in an image. The method includes: acquiring an image to be detected and a target character to be detected; acquiring image features based on the image to be detected; acquiring temporal visual features of the image to be detected based on the image to be detected and the image features; acquiring text enhancement features based on the image features and the target character; obtaining a segmentation map of the image to be detected based on the temporal visual features and the text enhancement features; and locating the position of the target character in the image to be detected according to the segmentation map.

[0006] In one implementation of the first aspect, obtaining the temporal visual features of the image to be detected based on the image to be detected and the image features includes: converting the image to be detected into a pixel matrix; extracting the temporal semantic information between each pixel in the pixel matrix to obtain a temporal semantic matrix; and performing cross-attention calculation on the temporal semantic matrix and the image features to obtain the temporal visual features.

[0007] In one implementation of the first aspect, cross-attention calculation of the temporal semantic matrix and the image features includes: using the temporal semantic matrix as a query sequence and the image features as a key sequence, querying the correlation between the temporal semantic matrix and the image features, and performing a weighted summation of the correlations to obtain the temporal visual features.

[0008] In one implementation of the first aspect, obtaining text enhancement features based on the image features and the target detected characters includes: setting the target detected characters according to a preset format to obtain preset input text; obtaining text features of the target detected characters based on the preset input text; and performing cross-attention calculation on the text features and the image features to obtain text enhancement features.

[0009] In one implementation of the first aspect, if there are several target detection characters, the text features of each target detection character are obtained sequentially, the text features of each target detection character are merged, and cross-attention calculation is performed on the merged text features and the image features to obtain text enhancement features.

[0010] In one implementation of the first aspect, obtaining a segmentation map of the image to be detected based on the temporal visual features and the text enhancement features includes: performing matrix addition on the temporal visual features and the text enhancement features to obtain a merged matrix; enhancing and fusing the merged matrix to obtain an output matrix; and processing the output matrix through a multilayer perceptron and an activation function to obtain a segmentation map.

[0011] In one implementation of the first aspect, locating the position of the target detection character in the image to be detected based on the segmentation map includes: converting the segmentation map into a pixel map with the same size as the image to be detected; determining whether the pixel value in the pixel map is greater than a preset threshold; if the pixel value in the pixel map is greater than the preset threshold, then the position of the pixel value is a text position; obtaining the position of the pixel value in the pixel map, wherein the position of the pixel value in the pixel map is the position of the target detection character in the image to be detected; if the pixel value in the pixel map is not greater than the preset threshold, then the position of the pixel value is a non-text position.

[0012] Secondly, this application provides a text location detection system in an image. The text location detection system includes: an acquisition module for acquiring an image to be detected and a target character to be detected, and acquiring image features based on the image to be detected; a first feature acquisition module for acquiring temporal visual features of the image to be detected based on the image to be detected and the image features; a second feature acquisition module for acquiring text enhancement features based on the image features and the target character to be detected; a feature processing module for obtaining a segmentation map of the image to be detected based on the temporal visual features and the text enhancement features; and a text location detection module for locating the position of the target character in the image to be detected according to the segmentation map.

[0013] Thirdly, this application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by an electronic device, implements the image text position detection method according to any one of the first aspects of this application.

[0014] Fourthly, this application provides an electronic device, comprising: a processor and a memory; the memory being used to store a computer program; the processor being used to execute the computer program stored in the memory, so that the electronic device performs the image text position detection method according to any one of the first aspects of this application.

[0015] According to the image text location detection method, system, medium, and electronic device of this application, the correlation between images and text is obtained by utilizing text characters and temporal semantics, and the information and connection between text, vision, and temporal sequence are strengthened by a large language model, thereby further performing text location detection and improving the positioning accuracy of text. Attached Figure Description

[0016] Figure 1 The diagram shown is a scene illustration of the electronic device of this application in one embodiment.

[0017] Figure 2 The diagram shown is a flowchart of an embodiment of the image text location detection method described in this application.

[0018] Figure 3 The diagram shown is a flowchart illustrating the acquisition of temporal visual features as described in this application in one embodiment.

[0019] Figure 4 The diagram shown is a flowchart illustrating the acquisition of text enhancement features as described in this application in one embodiment.

[0020] Figure 5 The diagram shown is a flowchart illustrating the process of acquiring text enhancement features as described in this application in another embodiment.

[0021] Figure 6 The diagram shown is a schematic flowchart of an embodiment of the method for obtaining a segmentation map of an image to be detected as described in this application.

[0022] Figure 7 The diagram shown is a flowchart illustrating the character position detection of the target as described in this application in one embodiment.

[0023] Figure 8 The diagram shown is a flowchart of another embodiment of the image text location detection method described in this application.

[0024] Figure 9 The diagram shown is a structural schematic of an embodiment of the image text location detection system described in this application.

[0025] Figure 10 The diagram shown is a structural schematic of the electronic device described in an embodiment of this application.

[0026] Component designation explanation

[0027] 11 cell phone 12 Tablet PC 13 laptop 900 Image text location detection system 910 Get Module 920 First Feature Acquisition Module 921 Second feature acquisition module 922 Feature processing module 930 Text position detection module 101 Processing unit 102 memory 1021 Random Access Memory 1022 Cache memory 1023 Storage System 1024 Programs / Utilities 1025 Program Module 103 bus 104 Input / output interface 105 Network adapter S1~S5 step S211~S213 step S221~S222 step S2211~S2213 step S31~S32 step Detailed Implementation

[0028] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0029] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. Therefore, the drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0030] Furthermore, the use of terms such as "first" and "second" in this application is for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of the various embodiments can be combined with each other, but only on the basis of being achievable by those skilled in the art. If the combination of technical solutions is contradictory or impossible to implement, such a combination of technical solutions should be considered non-existent and not within the scope of protection claimed in this application.

[0031] The following embodiments of this application provide a method and system for detecting text positions in images, a medium, and an electronic device. The method uses text characters and temporal semantics to obtain the correlation between images and text, and strengthens the fusion of information and connections between text, vision, and temporal information through a large language model, thereby further improving the accuracy of text position detection.

[0032] The image text location detection method of this application can be applied to, for example... Figure 1The electronic devices shown in this application may include mobile phones 11 with wireless charging capabilities, tablet computers 12, laptop computers 13, wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), etc. The specific types of electronic devices are not limited in this application embodiment.

[0033] For example, the electronic device may be a station (STAION, ST) in a WLAN with wireless charging capability, a cellular phone, cordless phone, Session Initiation Protocol (SIP) phone, Wireless Local Loop (WLL) station, Personal Digital Assistant (PDA) device, handheld device with wireless charging capability, computing device or other processing device, computer, laptop computer, handheld communication device, handheld computing device, and / or other devices for communication over a wireless system, as well as next-generation communication systems, such as mobile terminals in 5G networks, mobile terminals in future evolved Public Land Mobile Networks (PLMNs), or mobile terminals in future evolved Non-terrestrial Networks (NTNs).

[0034] For example, the electronic device can communicate with networks and other devices wirelessly. The wireless communication can use any communication standard or protocol, including but not limited to Global System for Mobile Communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), BT, GNSS, WLAN, NFC, FM, and / or IR technologies. The GNSS can include Global Positioning System (GPS), Global Navigation Satellite System (GLONASS), BeiDou Navigation Satellite System (BDS), Quasi-Zenith Satellite System (QZSS), and / or Satellite Based Augmentation Systems (SBAS).

[0035] The principles and implementation methods of the image text position detection method and system, medium and electronic device described in the embodiments of this application will be explained in detail below with reference to the accompanying drawings, so that those skilled in the art can understand the image text position detection method and system, medium and electronic device of this embodiment without creative effort.

[0036] Please see Figure 2 The diagram shows a flowchart illustrating the image text location detection method provided in this embodiment. Figure 2 As shown, the process includes steps S1 to S5.

[0037] Step S1: Obtain the image to be detected and the target character to be detected, and obtain image features based on the image to be detected.

[0038] Specifically, the target detection characters include Chinese and / or English; wherein, Chinese characters are single characters, and English characters are letters.

[0039] In one embodiment, the image to be detected is preprocessed, and the preprocessed image to be detected is input into a CLIPC image encoder to obtain the image features of the image to be detected.

[0040] For example, preprocessing the image to be detected includes: when using ResNet as the image encoder, the image size of the image to be detected needs to be adjusted to 224×224 pixels, and then the pixel values ​​of the image to be detected are converted from the range of [0, 255] to the range of [0, 1], and the average value of each color channel is subtracted and divided by the standard deviation.

[0041] Step S2: Obtain the temporal visual features of the image to be detected based on the image to be detected and the image features.

[0042] Specifically, the temporal semantic information between each pixel in the image to be detected is obtained based on the image to be detected, so as to further obtain a temporal semantic matrix, and then the temporal visual features of the image to be detected are obtained through the temporal semantic matrix and the image features.

[0043] In some implementations, obtaining the temporal visual features of the image to be detected based on the image to be detected and the image features includes: converting the image to be detected into a pixel matrix; extracting the temporal semantic information between each pixel in the pixel matrix to obtain a temporal semantic matrix; and performing cross-attention calculation on the temporal semantic matrix and the image features to obtain temporal visual features.

[0044] like Figure 3 As shown, obtaining temporal visual features includes the following steps S21 to S23.

[0045] Step S21: Convert the image to be detected into a pixel matrix.

[0046] Specifically, the image to be detected is divided into several pixels, each pixel having 3 channels, thereby obtaining the pixel matrix of the image to be detected.

[0047] Step S22: Extract the temporal semantic information between each pixel in the pixel matrix to obtain the temporal semantic matrix.

[0048] Specifically, the pixel matrix is ​​input into an LSTM (Long Short-Term Memory) network to extract the temporal semantic information between each pixel, thereby obtaining the temporal semantic matrix.

[0049] Step S23: Perform cross-attention calculation on the temporal semantic matrix and the image features to obtain temporal visual features.

[0050] In one embodiment, cross-attention calculation of the temporal semantic matrix and the image features includes: using the temporal semantic matrix as a query sequence and the image features as a key sequence, querying the correlation between the temporal semantic matrix and the image features, and performing a weighted summation of the correlations to obtain the temporal visual features.

[0051] It's important to note that cross-attention is a variant of the attention mechanism that allows the model to consider information from other sequences when processing sequential data. It works by calculating the similarity between each element in the query sequence and all elements in another sequence (or "key" sequence). Then, it performs a weighted sum of these similarities on the elements in the key sequence to obtain a weighted context vector. This vector is then combined with the elements from the query sequence to form the output.

[0052] Specifically, the temporal semantic matrix is ​​used as the query sequence (Query, Q), and the image features are used as keys (Key, K) and values ​​(Value, V). The matching degree between the query sequence and the key sequence is calculated to obtain a similarity matrix. The similarity matrix is ​​normalized using the softmax function to obtain an attention weight matrix. The values ​​(Value, V) are weighted and summed according to the attention weight matrix to obtain the output. The output is then incorporated into the query sequence as additional information to obtain the temporal visual features.

[0053] Step S3: Obtain text enhancement features based on the image features and the target detected characters.

[0054] In some implementations, obtaining text enhancement features based on the image features and the target detected characters includes: setting the target detected characters according to a preset format to obtain preset input text; obtaining text features of the target detected characters based on the preset input text; and performing cross-attention calculation on the text features and the image features to obtain text enhancement features.

[0055] like Figure 4 As shown, obtaining text enhancement features includes the following steps S31 to S33.

[0056] Step S31: Set the target detection characters according to the preset format to obtain the preset input text.

[0057] The preset format is: "image containing the character text"; the target detection character is written into the preset format to obtain the preset input text.

[0058] Step S32: Obtain the text features of the target detection character based on the preset input text.

[0059] Specifically, the preset input text is input into the CLIP text encoder to obtain the text features of the target detected character.

[0060] Step S33: Perform cross-attention calculation on the text features and the image features to obtain text enhancement features.

[0061] Specifically, the image features are used as the query sequence, and the text features are used as the key and value, thereby obtaining text enhancement features.

[0062] In some other implementations, if there are several target detection characters, the text features of each target detection character are obtained sequentially, the text features of each target detection character are merged, and cross-attention calculation is performed on the merged text features and the image features to obtain text enhancement features.

[0063] like Figure 5 As shown, several target detection characters are sequentially written into the preset format to obtain preset input text corresponding to the target detection characters. Each predicted input text is sequentially input into the CLIP text encoder, and the text features of each target detection character are obtained. The obtained text features are merged into a matrix, i.e., the merged text features. The merged text features are used as keys and values. The image features are used as a query sequence, and cross-attention calculation is performed on the image features and the merged text features to obtain the text enhancement features.

[0064] In this embodiment, N target detection characters are sequentially input into the CLIP text encoder according to a preset format to obtain N features with a shape of (1, 512). The N features with a shape of (1, 512) are merged to obtain text features with a shape of (N, 512).

[0065] Step S4: Obtain the segmentation map of the image to be detected based on the temporal visual features and the text enhancement features.

[0066] In some implementations, obtaining a segmentation map of the image to be detected based on the temporal visual features and the text enhancement features includes: performing matrix addition on the temporal visual features and the text enhancement features to obtain a merged matrix; enhancing and fusing the merged matrix to obtain an output matrix; and processing the output matrix using a multilayer perceptron and activation functions to obtain a segmentation map.

[0067] like Figure 6 As shown, obtaining the segmentation map of the image to be detected includes the following steps S41 to S43.

[0068] Step S41: Perform matrix addition on the temporal visual features and the text enhancement features to obtain a merged matrix.

[0069] Step S42: Strengthen the fusion of the merged matrix to obtain the output matrix.

[0070] Step S43: Process the output matrix using a multilayer perceptron and activation function to obtain a segmentation map.

[0071] Specifically, the temporal visual features and the text enhancement features are matrix-added to obtain a merged matrix. The merged matrix is ​​then input into a Large Language Model (LLM) to fuse the temporal visual feature information and text enhancement information within the merged matrix, resulting in an output matrix. The output matrix is ​​then processed by a multilayer perceptron and activation functions to further obtain a segmentation map of the image to be detected.

[0072] It's important to note that image segmentation is a technique and process that divides an image into regions with corresponding characteristics and extracts the target of interest. The purpose of image segmentation is to separate targets from the background in an image, facilitating further processing and analysis. Existing image segmentation methods can be categorized into several types: threshold-based segmentation methods, edge-based segmentation methods, and region-based segmentation methods. Threshold-based methods classify pixels in an image by setting one or more thresholds. This method is computationally simple and suitable for images with significant contrast between the background and foreground. Edge-based segmentation relies on detecting abrupt changes in image edges; commonly used edge detection operators include Sobel, Laplace, and Canny operators. Region-based methods, on the other hand, perform region growing or splitting and merging based on pixel similarity to form continuous regions.

[0073] In this embodiment, a pixel-based segmentation map of the image to be detected is obtained.

[0074] Step S5: Locate the position of the target detection character in the image to be detected based on the segmentation map.

[0075] Specifically, the acquired segmentation map is converted into a pixel map with the same size as the image to be detected. The pixel values ​​in the converted pixel map are located between [0, 1]. The size of the pixel value determines whether the pixel position contains the target detection character, and the position of the target detection character is located by the position of the pixel value.

[0076] In some implementations, locating the position of the target character in the image to be detected based on the segmentation map includes: converting the segmentation map into a pixel map with the same size as the image to be detected; determining whether the pixel value in the pixel map is greater than a preset threshold; if the pixel value in the pixel map is greater than the preset threshold, then the position of the pixel value is a text position; obtaining the position of the pixel value in the pixel map, and the position of the pixel value in the pixel map is the position of the target character in the image to be detected; if the pixel value in the pixel map is not greater than the preset threshold, then the position of the pixel value is a non-text position.

[0077] In one embodiment, the preset threshold is set to 0.5.

[0078] It should be noted that the preset threshold is not a unique fixed value and can be adjusted according to actual needs. However, since the pixel value in this embodiment is between [0, 1], the adjusted preset threshold is also between [0, 1].

[0079] like Figure 7 As shown, the segmentation map is converted into a pixel map with the same size as the image to be detected. The pixel values ​​in the pixel map are all between [0, 1]. It is determined whether the pixel values ​​in the pixel map are greater than 0.5.

[0080] If yes, then the position of the pixel value in the image to be detected is the text position, and the position of the pixel value in the image to be detected is the position of the target detected character in the image to be detected; if no, then the position of the pixel value in the image to be detected is a non-text position.

[0081] Please see Figure 8 The above is a flowchart of another embodiment of the image text location detection method described in this application.

[0082] like Figure 8 As shown, the image text location detection method includes:

[0083] Obtain the image to be detected and the target characters to be detected: The size of the image to be detected is (h, w), where h and w represent the height and width of the image to be detected, respectively; there are N target characters to be detected.

[0084] Image feature acquisition: The image to be detected is input into the CLIP image encoder to acquire the image features, wherein the shape of the image features is (1, 512).

[0085] Obtaining the temporal semantic matrix: The image to be detected is input into the pixel embedding layer to obtain the temporal semantic matrix; specifically: the image to be detected is split into M pixels, each pixel has 3 channels, thus obtaining a pixel matrix of shape (M, 3), where the number of pixels M is the product of the height and width of the image to be detected, i.e., the product of h and w. The obtained pixel matrix is ​​then processed through an LSTM (Long Short-Term Memory) network to extract temporal semantic information between pixels, thereby obtaining the temporal semantic matrix, wherein the shape of the temporal semantic matrix is ​​(M, 512).

[0086] Temporal visual features are obtained based on the temporal semantic matrix and the image features: the temporal semantic matrix is ​​used as the query sequence Q, and the image features are used as the key K and value V. These are input into a cross-attention network for calculation, thereby querying the correlation between the temporal semantic matrix and the image features and performing a weighted summation to obtain a temporal visual feature with shape (M, 512). It should be noted that in the cross-attention network, the shape of the output feature is the same as the shape of the input query sequence.

[0087] Obtaining text features: The N target detection characters obtained are sequentially input into the CLIP text encoder in the format of "image containing "character" text, thereby obtaining N features with shape (1, 512). The N features with shape (1, 512) are merged to obtain text features with shape (N, 512).

[0088] Based on the image features and the text features, text enhancement features are obtained: the image features are used as the query sequence Q, and the text features are used as the key K and value V. The data are input into a cross-attention network for calculation, thereby querying the correlation between the image features and the text features and performing a weighted summation to obtain text enhancement features with shape (1, 512).

[0089] The segmentation map is obtained based on the temporal visual features and the text enhancement features: the temporal visual features and the text enhancement features are matrix-added, and the temporal visual feature information and text enhancement information are enhanced and fused through a large language model (LLM) to obtain an output matrix of shape (M, 512). The pure output matrix is ​​processed through a multilayer perceptron and activation function to obtain a segmentation map of shape (M, 1).

[0090] Based on the segmentation map, the position of the target character in the image to be detected is located: the segmentation map is converted into a pixel map of shape (h, w), where h and w represent the height and width of the pixel map, respectively, and the pixel map contains... Each pixel value is located between [0, 1]. The pixel value in the pixel image is determined to be greater than 0.5. If the pixel value is greater than 0.5, the position of the pixel value is considered a text position. The position of the pixel value in the pixel image is obtained, and this position is the position of the target character in the image to be detected. If the pixel value is not greater than 0.5, the position of the pixel value is considered a non-text position.

[0091] It should be noted that the protection scope of the image text position detection method described in this application is not limited to the execution order of the steps listed in this embodiment. Any solution implemented by adding, subtracting, or replacing steps in the prior art based on the principles of this application is included within the protection scope of this application.

[0092] Please see Figure 9 The image shown is a schematic diagram of the text location detection system described in this application embodiment.

[0093] like Figure 9 As shown, the image text location detection system 900 includes: an acquisition module 910, a first feature acquisition module 920, a second feature acquisition module 930, a feature processing module 940, and a text location detection module 950.

[0094] The acquisition module 910 is used to acquire the image to be detected and the target character to be detected, and to acquire image features based on the image to be detected;

[0095] The first feature acquisition module 920 is used to acquire temporal visual features of the image to be detected based on the image to be detected and the image features;

[0096] The second feature acquisition module 930 is used to acquire text enhancement features based on the image features and the target detected characters;

[0097] The feature processing module 940 is used to obtain a segmentation map of the image to be detected based on the temporal visual features and the text enhancement features;

[0098] The text position detection module 950 is used to locate the position of the target character in the image to be detected based on the segmentation map.

[0099] It should be noted that the way each module of the image text position detection system described in this embodiment implements its corresponding function corresponds one-to-one with the image text position detection method described in this application embodiment, so it will not be repeated here.

[0100] It should be noted that the above division of modules is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software via processing element calls; they can be fully implemented in hardware; or some modules can be implemented by processing element calls to software, while others are implemented in hardware. For example, module x can be a separate processing element, or it can be integrated into a chip in the aforementioned device. Alternatively, it can be stored as program code in the memory of the aforementioned device, and its function can be called and executed by a processing element of the device. The implementation of other modules is similar. Moreover, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element mentioned here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0101] For example, these modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more Digital Signal Processors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs). As another example, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together to form a system-on-a-chip (SOC).

[0102] This application also provides a computer-readable storage medium. Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing a processor. The program can be stored in a computer-readable storage medium, which is a non-transitory medium, such as random access memory, read-only memory, flash memory, hard disk, solid-state drive, magnetic tape, floppy disk, optical disk, and any combination thereof. The storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DVD)), or a semiconductor medium (e.g., solid-state drive (SSD)).

[0103] This application also provides an electronic device, including a processor and a memory.

[0104] Specifically, a memory is used to store computer programs; memory includes various media that can store program code, such as ROM, RAM, magnetic disks, USB flash drives, memory cards, or optical discs.

[0105] The processor executes a computer program stored in memory to enable the electronic device to perform the aforementioned method for detecting the location of text in an image.

[0106] like Figure 10 As shown, the electronic device of this application is embodied in the form of a general-purpose computing device. The components of the electronic device may include, but are not limited to: one or more processors or processing units 101, memory 102, and bus 103 connecting different system components (including memory 102 and processing unit 101).

[0107] Bus 103 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. For example, these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0108] Electronic devices typically include a variety of computer-readable media. These media can be any available media that can be accessed by the electronic device, including volatile and non-volatile media, and removable and non-removable media.

[0109] Memory 102 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 1021 and / or cache memory 1022. The electronic device may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 1023 may be used to read and write non-removable, non-volatile magnetic media (… Figure 10 Not shown; usually referred to as a "hard drive"). Although Figure 10 As not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 103 via one or more data media interfaces. Memory 102 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this application.

[0110] A program / utility 1024 having a set (at least one) of program modules 1025 may be stored, for example, in memory 102. Such program modules 1025 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 1025 typically perform the functions and / or methods described in the embodiments of this application.

[0111] The electronic device can also communicate with one or more external devices (e.g., keyboard, pointing device, display, etc.), and with one or more devices that enable a user to interact with the electronic device, and / or with any device that enables the electronic device to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed through input / output (I / O) interface 104. Furthermore, the electronic device can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) through network adapter 105. Figure 10As shown, network adapter 105 communicates with other modules of the electronic device via bus 103. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0112] The descriptions of the processes or structures corresponding to the above figures each have their own emphasis. For parts of a process or structure that are not described in detail, please refer to the relevant descriptions of other processes or structures.

[0113] In summary, the image text location detection method, system, medium, and electronic device of this application utilize text characters and temporal semantics to obtain the correlation between images and text, and strengthens the fusion of information and connections between text, vision, and temporal sequences through a large language model, thereby further improving text location detection accuracy. Therefore, this application effectively overcomes the various shortcomings of existing technologies and has high industrial applicability.

[0114] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. A method for detecting the position of text in an image, characterized in that, The method includes: Acquire the image to be detected and the target character to be detected, and obtain image features based on the image to be detected; The temporal visual features of the image to be detected are obtained based on the image to be detected and the image features. Text enhancement features are obtained based on the image features and the detected target characters; The segmentation map of the image to be detected is obtained based on the temporal visual features and the text enhancement features; The position of the target character in the image to be detected is located based on the segmentation map.

2. The method for detecting the position of text in an image according to claim 1, characterized in that, Obtaining the temporal visual features of the image to be detected based on the image to be detected and the image features includes: The image to be detected is converted into a pixel matrix; Extract the temporal semantic information between each pixel in the pixel matrix to obtain the temporal semantic matrix; Cross-attention calculation is performed on the temporal semantic matrix and the image features to obtain temporal visual features.

3. The method for detecting the position of text in an image according to claim 2, characterized in that, The cross-attention calculation of the temporal semantic matrix and the image features includes: using the temporal semantic matrix as a query sequence and the image features as a key sequence, querying the correlation between the temporal semantic matrix and the image features, and performing a weighted summation of the correlations to obtain the temporal visual features.

4. The method for detecting the position of text in an image according to claim 1, characterized in that, The text enhancement features obtained based on the image features and the detected target characters include: The target detection characters are set according to a preset format to obtain preset input text; The text features of the target detected characters are obtained based on the preset input text; Cross-attention calculation is performed on the text features and the image features to obtain text enhancement features.

5. The method for detecting the position of text in an image according to claim 4, characterized in that, If there are several target detection characters, the text features of each target detection character are obtained sequentially, the text features of each target detection character are merged, and cross-attention calculation is performed on the merged text features and the image features to obtain text enhancement features.

6. The method for detecting the position of text in an image according to claim 1, characterized in that, The segmentation map of the image to be detected, obtained based on the temporal visual features and the text enhancement features, includes: Perform matrix addition on the temporal visual features and the text enhancement features to obtain a merged matrix; The merged matrices are enhanced to obtain the output matrix; The output matrix is ​​processed using a multilayer perceptron and activation functions to obtain a segmentation map.

7. The method for detecting the position of text in an image according to claim 1, characterized in that, Locating the position of the target character in the image to be detected based on the segmentation map includes: The segmentation map is converted into a pixel map with the same size as the image to be detected; Determine whether the pixel value in the pixel image is greater than a preset threshold; If the pixel value in the pixel image is greater than the preset threshold, then the position of the pixel value is the text position; the position of the pixel value in the pixel image is obtained, and the position of the pixel value in the pixel image is the position of the target detected character in the image to be detected; If the pixel value in the pixel image is not greater than the preset threshold, then the location of the pixel value is a non-text location.

8. A text location detection system in an image, characterized in that, The system includes: The acquisition module is used to acquire the image to be detected and the target character to be detected, and to acquire image features based on the image to be detected; The first feature acquisition module is used to acquire the temporal visual features of the image to be detected based on the image to be detected and the image features. The second feature acquisition module is used to acquire text enhancement features based on the image features and the target detected characters; The feature processing module is used to obtain a segmentation map of the image to be detected based on the temporal visual features and the text enhancement features; The text position detection module is used to locate the position of the target character in the image to be detected based on the segmentation map.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by an electronic device, the program implements the image text location detection method as described in any one of claims 1 to 7.

10. An electronic device, characterized in that, Including processor and memory; The memory is used to store computer programs; The processor is used to execute the computer program stored in the memory to cause the electronic device to perform the image text location detection method according to any one of claims 1 to 7.

Images