Text detection model training method based on visual prompt and text detection method

Through the text detection model training method based on visual cues, the text detection model is trained using sample images of multiple existing scenarios, which solves the problem of insufficient sample images in industrial scenarios, realizes text detection suitable for industrial scenarios, and improves the model's Generalization capability and detection efficiency.

CN118968518APending Publication Date: 2024-11-15HANGZHOU HIKROBOT TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411008089.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-25
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

In industrial scenarios, since the number of sample images is small, it is difficult to obtain the scene image of the industrial scenario to be detected as the sample image, which makes it difficult to train the text detection model.

Method used

Using the text detection model training method based on visual cues, the sample images of multiple existing scenes are obtained, the visual cues characteristics of image and text area are extracted, and the position information of the text area is predicted, and the model is adjusted based on the label information. Parameters to realize the training of text detection model.

Benefits of technology

When the scene image of the scene to be detected is not relied on as the sample image, a text detection model suitable for industrial scenarios is trained, which improves the generalization ability of the model, reduces the risk of missed detection and overfitting, and improves the detection efficiency .

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118968518A_ABST
    Figure CN118968518A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides a training method of a text detection model based on visual prompt and a text detection method, and relates to the technical field of image processing. The training method comprises the steps of obtaining a sample image set; extracting image features of the sample image, and determining visual prompt features of a text region in the sample image; for each sample image, inputting the image feature and the visual prompt feature of each text region into a text detection model, so that the text detection model fuses the image feature and the visual prompt feature of each text region, and based on the fused feature, performing prediction by using a position detector to obtain a prediction result corresponding to the sample image; and performing model parameter adjustment on the text detection model based on the label information of each sample image and the corresponding prediction result. By applying the scheme provided by the embodiment of the invention, the text detection model suitable for the to-be-detected industrial scene can be obtained through training on the premise of not depending on the scene image of the to-be-detected industrial scene as the sample image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a training method for a text detection model based on visual cues and a text detection method. Background Art

[0002] Currently, in many industrial scenarios, it is necessary to detect specific text in the collected scene images, for example, detecting the production date marked on the products in the production workshop, etc., in order to obtain the position of the specific text in the scene image, or the character content of the specific text.

[0003] Generally, in the related art, in order to realize the detection of specific text in the scene images of industrial scenes, the scene images in the industrial scenes can be collected as sample images, and the specific text in the sample images can be used as labels of the sample images. Then, the model is trained using the labeled sample images to obtain a text detection model for detecting specific text in the scene images of industrial scenes.

[0004] However, since the diversity of industrial scenes is lower than that of natural scenes, the number of sample images obtained is small, and in some cases, it is even impossible to obtain scene images of the industrial scenes to be detected as sample images. Therefore, a training method for a text detection model is urgently needed to train a text detection model suitable for the industrial scenes to be detected without relying on the scene images of the industrial scenes to be detected as sample images. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a training method and a text detection method based on visual cues for a text detection model, so as to train a text detection model suitable for the industrial scene to be detected without relying on the scene image of the industrial scene to be detected as a sample image. The specific technical solution is as follows:

[0006] In a first aspect, an embodiment of the present application provides a method for training a text detection model based on visual cues, the method comprising:

[0007] Acquire a sample image set; wherein the sample image set includes a plurality of sample images obtained by collecting images of a plurality of existing scenes, the types of text areas in the plurality of sample images include a plurality of preset text types, and each sample image has: label information at least including position information of the text area in the sample image;

[0008] Extracting image features of each sample image, and determining visual cue features of each text region in each sample image; wherein the visual cue features of each text region are used to characterize: the attributes of the region under the text type to which the text region belongs;

[0009] For each sample image, the image features of the sample image and the visual cue features of each text area in the sample image are respectively input into the text detection model to be trained, so that the text detection model performs a fusion operation on the image features of the sample image and the visual cue features of each text area in the sample image, and based on the fusion features obtained by the fusion operation, a position detector is used to perform a prediction analysis to obtain a prediction result corresponding to the sample image; wherein the prediction result at least includes: the position information of the text area in the sample image;

[0010] Based on the label information of each sample image and the corresponding prediction result, the model parameters of the text detection model are adjusted.

[0011] Optionally, in a specific implementation, before inputting the image features of each sample image and the visual cue features of each text area in the sample image into the text detection model to be trained, the method further includes:

[0012] Acquire text prompt information of each text region in each sample image, and determine a text prompt feature of the text prompt information of each text region as the text prompt feature of the text region;

[0013] For each sample image, the image features of the sample image and the visual cue features of each text area in the sample image are respectively input into the text detection model to be trained, so that the text detection model performs a fusion operation on the image features of the sample image and the visual cue features of each text area in the sample image, including:

[0014] For each sample image, the image features of the sample image, the visual prompt features of each text area in the sample image, and the text prompt features of each text area in the sample image are respectively input into the text detection model to be trained, so that the text detection model performs a multimodal feature fusion operation on the image features of the sample image and the target prompt features of each text area in the sample image; wherein the target prompt features of each text area include: the visual prompt features and the text prompt features of the text area.

[0015] Optionally, in a specific implementation, the label information of each sample image further includes: character content of a text area in the sample image;

[0016] The text detection model performs prediction analysis based on the fusion features obtained by the fusion operation using the position detector to obtain a prediction result corresponding to the sample image, including:

[0017] The text detection model performs prediction analysis based on the fusion features obtained by the fusion operation using a position detector and a character recognizer to obtain a prediction result corresponding to the sample image; wherein the prediction result also includes: the character content of the text area in the sample image.

[0018] Optionally, in a specific implementation, obtaining text prompt information of each text area in each sample image includes:

[0019] Obtaining character content of each text area in each sample image as text prompt information of each text area;

[0020] or,

[0021] An expression of the character content of each text area in each sample image is obtained as text prompt information of each text area.

[0022] Optionally, in a specific implementation, the method for collecting the multiple sample images includes:

[0023] Scene images of various existing industrial scenes are collected as multiple sample images.

[0024] Optionally, in a specific implementation, determining the visual cue features of each text area in each sample image includes:

[0025] A target text type related to a target industrial scene is determined, and visual cue features of each text region belonging to the target text type in each sample image are determined.

[0026] In a second aspect, an embodiment of the present application provides a text detection method based on visual cues, the method comprising:

[0027] Acquire an image to be detected in a target industrial scene; wherein the text type of the text area in the image to be detected is: at least one of a plurality of preset text types;

[0028] The image to be detected is input into a pre-trained text detection model to obtain a detection result; wherein the text detection model is trained based on any one of the methods provided in the first aspect above.

[0029] In a third aspect, an embodiment of the present application provides a training device for a text detection model, the device comprising:

[0030] A sample image acquisition module, used to acquire a sample image set; wherein the sample image set includes a plurality of sample images obtained by collecting images of a plurality of existing scenes, the types of text areas in the plurality of sample images include a plurality of preset text types, and each sample image has: label information at least including position information of the text area in the sample image;

[0031] The first feature extraction module is used to extract the image features of each sample image and determine the visual cue features of each text area in each sample image; wherein the visual cue features of each text area are used to characterize: the attributes of the area under the text type to which the text area belongs;

[0032] A feature fusion module is used to input the image features of each sample image and the visual cue features of each text area in the sample image into the text detection model to be trained, so that the text detection model performs a fusion operation on the image features of the sample image and the visual cue features of each text area in the sample image, and performs a prediction analysis based on the fusion features obtained by the fusion operation using a position detector to obtain a prediction result corresponding to the sample image; wherein the prediction result at least includes: the position information of the text area in the sample image;

[0033] The parameter adjustment module is used to adjust the model parameters of the text detection model based on the label information of each sample image and the corresponding prediction result.

[0034] Optionally, in a specific implementation, the device further includes:

[0035] A second feature extraction module is used for obtaining text prompt information of each text area in each sample image and determining text prompt features of the text prompt information of each text area as the text prompt features of the text area before inputting the image features of the sample image and the visual prompt features of each text area in the sample image into the text detection model to be trained for each sample image;

[0036] The feature fusion module is specifically used to input the image features of the sample image, the visual prompt features of each text area in the sample image, and the text prompt features of each text area in the sample image into the text detection model to be trained for each sample image, so that the text detection model performs a multimodal feature fusion operation on the image features of the sample image and the target prompt features of each text area in the sample image; wherein the target prompt features of each text area include: the visual prompt features and the text prompt features of the text area;

[0037] Optionally, in a specific implementation, the label information of each sample image further includes: character content of a text area in the sample image;

[0038] The feature fusion module is specifically used to input the image features of the sample image, the visual prompt features of each text area in the sample image, and the text prompt features of each text area in the sample image into the text detection model to be trained for each sample image, so that the text detection model performs a multimodal feature fusion operation on the image features of the sample image and the target prompt features of each text area in the sample image, and performs prediction analysis based on the fusion features obtained by the fusion operation using a position detector and a character recognizer to obtain a prediction result corresponding to the sample image; wherein the prediction result also includes: the character content of the text area in the sample image;

[0039] Optionally, in a specific implementation, the second feature extraction module is specifically used to:

[0040] Obtaining character content of each text area in each sample image as text prompt information of each text area;

[0041] or,

[0042] Obtaining the expression of the character content of each text area in each sample image as the text prompt information of each text area;

[0043] Optionally, in a specific implementation, the device further includes:

[0044] An image acquisition module, used for acquiring scene images of various existing industrial scenes as multiple sample images;

[0045] Optionally, in a specific implementation, the first feature extraction module is specifically used to:

[0046] A target text type related to a target industrial scene is determined, and visual cue features of each text region belonging to the target text type in each sample image are determined.

[0047] In a fourth aspect, an embodiment of the present application provides a text detection device based on visual cues, the device comprising:

[0048] An image acquisition module is used to acquire an image to be detected in a target industrial scene; wherein the text type of the text area in the image to be detected is: at least one of a plurality of preset text types;

[0049] A result detection module is used to input the image to be detected into a pre-trained text detection model to obtain a detection result; wherein the text detection model is trained based on any one of the methods provided in the first aspect above.

[0050] In a fifth aspect, an embodiment of the present application provides an electronic device, including:

[0051] Memory, used to store computer programs;

[0052] The processor is used to implement any of the methods provided in the first aspect and / or any of the methods provided in the second aspect when executing the program stored in the memory.

[0053] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, implements any of the methods provided in the first aspect above, and / or implements any of the methods provided in the second aspect above.

[0054] In a seventh aspect, an embodiment of the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the methods provided in the first aspect above, and / or implement any of the methods provided in the second aspect above.

[0055] Beneficial effects of the embodiments of the present application:

[0056] It can be seen from the above that, when applying the solution provided by the embodiment of the present application, when training a text detection model, the training data used are the image features of the sample image and the visual cue features of each text area in the sample image, wherein the sample image is obtained by image acquisition of a variety of existing scenes, and the label information of each sample image includes at least the position information of the text area in the sample image, and the types of text areas in the multiple sample images include a variety of preset text types. During the training process, the text detection model can predict the position information of the text area in the sample image based on the fusion features of the image features of the sample image and the visual cue features of the text area, and then, the model parameters of the text detection model can be adjusted based on the label information of the sample image and the corresponding prediction results.

[0057] Based on this, since in the model training process, the fusion features learned by the text detection model come from multiple sample images collected under various existing scenes, and multiple text areas belonging to each preset text type in the above multiple sample images, the text detection model can learn the correlation characteristics between the text areas of the preset text types under the scene and the scene and the characteristics of the text area itself during training, so that the text detection model obtained by training has a higher generalization ability, and therefore, the trained text detection model can detect the text areas of each preset text type in the image outside the sample image set. That is, the trained text detection model can detect the text areas of each preset text type in the scene image of the scene to be detected without relying on the scene image of the scene to be detected as a sample image. In this way, the scheme provided by the application embodiment can be realized without relying on the scene image of the industrial scene to be detected as a sample image, and the text detection model suitable for the industrial scene to be detected can be trained.

[0058] In addition, since the sample images in the sample image set can come from a variety of existing scenes, rather than being limited to industrial scenes, a large number of sample images can be obtained for training the text detection model. Based on this, using a large number of sample images to train the text detection model can avoid overfitting of the trained text detection model, reduce the missed detection rate of the trained text detection model, and improve the detection efficiency of the trained text detection model.

[0059] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0061] Figure 1 A flowchart of a method for training a text detection model based on visual cues provided in an embodiment of the present application;

[0062] Figure 2 A flowchart of another method for training a text detection model based on visual cues provided in an embodiment of the present application;

[0063] Figure 3 A flowchart of another method for training a text detection model based on visual cues provided in an embodiment of the present application;

[0064] Figure 4 A flowchart of another method for training a text detection model based on visual cues provided in an embodiment of the present application;

[0065] Figure 5 The above-mentioned embodiments provided in this application Figure 4 The model reasoning flow chart of the specific implementation method shown;

[0066] Figure 6 The above-mentioned embodiments provided in this application Figure 4 A structural diagram of a text detection model of a specific implementation method shown;

[0067] Figure 7 A flowchart of a text detection method based on visual cues provided in an embodiment of the present application;

[0068] Figure 8 A schematic diagram of the structure of a training device for a text detection model based on visual cues provided in an embodiment of the present application;

[0069] Fig. 9 A schematic diagram of the structure of a text detection device based on visual cues provided in an embodiment of the present application;

[0070] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0071] Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0072] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0073] Since the diversity of industrial scenes is lower than that of natural scenes, the number of sample images obtained is small, and in some cases, it is even impossible to obtain scene images of the industrial scenes to be detected as sample images. Therefore, a training method for a text detection model is urgently needed to train a text detection model suitable for the industrial scenes to be detected without relying on the scene images of the industrial scenes to be detected as sample images.

[0074] In order to solve the above technical problems, an embodiment of the present application provides a training method for a text detection model based on visual cues.

[0075] Among them, the method can be applied to various application scenarios that require text detection on scene images, such as industrial scenarios, etc.; and the method can be applied to various electronic devices with data processing capabilities, such as laptops, desktop computers, etc., and the electronic device can be an independent electronic device or a device cluster composed of multiple electronic devices, hereinafter referred to as the electronic device. Based on this, the embodiment of the present application does not specifically limit the application scenario and execution subject of the method.

[0076] The present invention provides a method for training a text detection model based on visual cues, which may include the following steps:

[0077] Acquire a sample image set; wherein the sample image set includes a plurality of sample images obtained by collecting images of a plurality of existing scenes, the types of text areas in the plurality of sample images include a plurality of preset text types, and each sample image has: label information at least including position information of the text area in the sample image;

[0078] Extracting image features of each sample image, and determining visual cue features of each text region in each sample image; wherein the visual cue features of each text region are used to characterize: the attributes of the region under the text type to which the text region belongs;

[0079] For each sample image, the image features of the sample image and the visual cue features of each text area in the sample image are respectively input into the text detection model to be trained, so that the text detection model performs a fusion operation on the image features of the sample image and the visual cue features of each text area in the sample image, and based on the fusion features obtained by the fusion operation, a position detector is used to perform a prediction analysis to obtain a prediction result corresponding to the sample image; wherein the prediction result at least includes: the position information of the text area in the sample image;

[0080] Based on the label information of each sample image and the corresponding prediction result, the model parameters of the text detection model are adjusted.

[0081] It can be seen from the above that, when applying the solution provided by the embodiment of the present application, when training a text detection model, the training data used are the image features of the sample image and the visual cue features of each text area in the sample image, wherein the sample image is obtained by image acquisition of a variety of existing scenes, and the label information of each sample image includes at least the position information of the text area in the sample image, and the types of text areas in the multiple sample images include a variety of preset text types. During the training process, the text detection model can predict the position information of the text area in the sample image based on the fusion features of the image features of the sample image and the visual cue features of the text area, and then, the model parameters of the text detection model can be adjusted based on the label information of the sample image and the corresponding prediction results.

[0082] Based on this, since in the model training process, the fusion features learned by the text detection model come from multiple sample images collected under various existing scenes, and multiple text areas belonging to each preset text type in the above multiple sample images, the text detection model can learn the correlation characteristics between the text areas of the preset text types under the scene and the scene and the characteristics of the text area itself during training, so that the text detection model obtained by training has a higher generalization ability, and therefore, the trained text detection model can detect the text areas of each preset text type in the image outside the sample image set. That is, the trained text detection model can detect the text areas of each preset text type in the scene image of the scene to be detected without relying on the scene image of the scene to be detected as a sample image. In this way, the scheme provided by the application embodiment can be realized without relying on the scene image of the industrial scene to be detected as a sample image, and the text detection model suitable for the industrial scene to be detected can be trained.

[0083] In addition, since the sample images in the sample image set can come from a variety of existing scenes, rather than being limited to industrial scenes, a large number of sample images can be obtained for training the text detection model. Based on this, using a large number of sample images to train the text detection model can avoid overfitting of the trained text detection model, reduce the missed detection rate of the trained text detection model, and improve the detection efficiency of the trained text detection model.

[0084] Below, in conjunction with the accompanying drawings, a training method for a text detection model based on visual cues provided in an embodiment of the present application is specifically described.

[0085] Figure 1 A flowchart of a method for training a text detection model based on visual cues provided in an embodiment of the present application is shown below. Figure 1 As shown, the method may include the following steps S101-S105.

[0086] S101: Obtain a sample image set.

[0087] Among them, the sample image set includes multiple sample images obtained by image acquisition of multiple existing scenes, the types of text areas in the multiple sample images include multiple preset text types, and each sample image has: label information that at least includes location information of the text area in the sample image.

[0088] Usually, when training a model, it is necessary to first obtain a training set. Therefore, in the embodiment of the present application, when training a text detection model, an image sample set may be first obtained.

[0089] Optionally, in a specific implementation, since it is impossible to obtain a scene image of the scene to be detected as a sample image in many cases, for example, an industrial scene to be detected, the scene to be detected can be regarded as a non-existing scene, and the scene whose scene image can be collected can be regarded as an existing scene. In this way, image collection can be performed on a variety of existing scenes, and the multiple scene images obtained by collection can be used as sample images in the above-mentioned sample image set. The above-mentioned existing scene can be any scene other than the scene to be detected. For example, for an industrial scene to be detected, the above-mentioned existing scene can be other industrial scenes other than the industrial scene to be detected, or can be other scenes other than industrial scenes, such as natural scenes. In this way, the above-mentioned image sample set can include multiple sample images obtained by image collection of a variety of existing scenes.

[0090] The so-called scenarios to be detected are scenarios in which the text detection model is required to be used subsequently, that is, scenarios to which the trained text detection model is applicable.

[0091] Optionally, in another specific implementation, for the scene to be detected, only a small number of sample images may be collected. In this case, the scene to be detected may also be used as an existing scene. In this way, when performing image acquisition on the existing scene to obtain sample images and then obtaining a sample image set, the above-mentioned multiple existing scenes may include the above-mentioned scene to be detected. Therefore, the sample images included in the acquired sample image set may include scene images obtained by performing image acquisition on the above-mentioned scene to be detected, which is also reasonable.

[0092] In summary, the purpose of this application is to decouple the sample images that the model training process relies on from the industrial scenes that the model is adapted to, that is, the model training process does not need to completely rely on the sample images of the scene to be detected. Then, the industrial scene to be detected can be a scene different from the above-mentioned multiple existing scenes, or it can be one of the multiple existing scenes, which is reasonable.

[0093] That is to say, in the prior art, since the diversity of actual industrial scenes is not as rich as that of natural scenes, if the text localization network is fine-tuned based on the small amount of labeled data provided, although it is easy to learn, it is easy to cause network overfitting. Therefore, in the embodiment of the present application, in industrial scenes, the diversity of publicly available and general industrial scene image data can be fully utilized to locate and identify specific text in industrial specific scenes, thereby avoiding overfitting of the trained text detection model, reducing the missed detection rate of the trained text detection model, and improving the detection efficiency of the trained text detection model.

[0094] Optionally, in a specific implementation, the method for collecting the multiple sample images in the above-mentioned image sample set may include the following step 1011:

[0095] Step 1011: Collect scene images of various existing industrial scenes as multiple sample images.

[0096] In this specific implementation, usually, when the scene to be detected is an industrial scene, it is often impossible to obtain scene images of the industrial scene to be detected as sample images, or only a small number of images of the industrial scene to be detected can be obtained as sample images. Therefore, multiple existing industrial scenes can be used as sources of sample images, and then, scene images of multiple existing industrial scenes can be collected as multiple sample images. For example, image collection can be performed at the scene of various existing industrial scenes, so that scene images of the real scene of the existing industrial scene can be collected as sample images.

[0097] In addition, the above-mentioned sample images may include text areas. The types of text areas included in different sample images may be the same or different. The same sample image may also include multiple different types of text areas. Therefore, the types of text areas in the multiple sample images included in the above-mentioned sample image set may be multiple. For example, the type of text area in sample image A in the sample image set is text type 1, and the type of text area in sample image B in the sample image set is text type 2 and 3, etc. In this way, the types of text areas in the multiple sample images included in the above-mentioned sample image set include multiple preset text types. The preset text types may be all types of text areas in the above-mentioned multiple sample images, or may be multiple specific text types set according to the detection needs of the scenes to be detected to which the text detection model may be applied.

[0098] The above text types can be divided according to the meaning represented by the character content in the text area, for example, product date text in an industrial scenario, product number text in an industrial scenario, etc., or can be divided according to the language type to which the characters in the text area belong, for example, all-English text, all-Chinese text, all-digital text, Chinese and digital mixed text, etc. In this regard, the embodiments of the present application do not limit the specific division method of the above text types.

[0099] In order to realize the training of the text detection model, label information can usually be set for each sample image. Considering that the embodiment of the present application trains a text detection model for detecting text in an image, the position information of the text in the image detected can usually be used as the detection result of the text detection model. Based on this, when setting label information for each of the above sample images, the label information possessed by each sample image can at least include: label information of the position information of the text area in the sample image. In this way, during the model training process, the model parameters of the text detection model can be adjusted based on the difference between the position information of the text area in the prediction result of the text detection model for each sample image and the label information of the sample image, so as to realize the training of the text detection model.

[0100] S102: extracting image features of each sample image;

[0101] S103: Determine the visual cue features of each text area in each sample image.

[0102] The visual prompt feature of each text region is used to represent: the attribute of the region under the text type to which the text region belongs.

[0103] After obtaining the above-mentioned sample image set, for each sample image in the above-mentioned sample image set, the image features of the sample image can be first extracted. For example, image encoders such as ViT (Vision Transformer, a visual-based time series model), ResNet (Deep Residual Networks), and ConvNeXt (pure convolutional network model) can be used to extract the image features of the sample image.

[0104] For each text area in each sample image, the visual cue features of each text area in the sample image can be determined, and the visual cue features of each text area are used to characterize the attributes of the area under the text type to which the text area belongs, such as area shape, area size, area aspect ratio, text voice type, text length, etc.

[0105] The visual cue features of each of the above-mentioned text regions may be determined in a variety of ways. For example, each text region may be captured, and then a regional feature map of each text region may be extracted using a visual cue device as the visual cue feature of each text region.

[0106] Optionally, in a specific implementation, such as Figure 2 As shown, in the above step S102, the step of determining the visual cue features of each text area in each sample image may include the following steps S1021-S1022:

[0107] S1021: Determine the target text type related to the target industrial scenario;

[0108] S1022: Determine visual cue features of each text region belonging to the target text type in each sample image.

[0109] In this specific implementation, usually, although there are differences in the specific scene details of different industrial scenes, the types of text areas in the scene images of different industrial scenes are often the same, or relatively similar. Therefore, when training the above-mentioned text detection model, in order to reduce training data, improve model training efficiency and reduce resource consumption of model training, the target industrial scenes to which the text detection model may be applied can be estimated, such as product production scenes, product inspection scenes, product warehousing scenes, etc., so that the target text types related to the above-mentioned target industrial scenes can be further determined, such as product production date, product production number, etc. In this way, when determining the visual cue features of the text area, the visual cue features of each text area belonging to the above-mentioned target text type in each sample image can be determined, without determining the visual cue features of other types of text areas other than the above-mentioned target text type.

[0110] For example, the determined target text types include: product production date and product production number, and a sample image includes three text areas, among which text area 1 is the product production date, text area 2 is the product manufacturer, and text area 3 is the product production number. For this sample image, the visual cue features of text area 1 and text area 3 can be determined, without determining the visual cue features of text area 2.

[0111] That is to say, in this specific implementation, for each sample image, the visual cue features input into the text detection model to be trained are: the visual cues of each text area belonging to the target text type in the sample image.

[0112] S104: For each sample image, the image features of the sample image and the visual cue features of each text area in the sample image are respectively input into the text detection model to be trained, so that the text detection model performs a fusion operation on the image features of the sample image and the visual cue features of each text area in the sample image, and based on the fusion features obtained by the fusion operation, a position detector is used to perform prediction analysis to obtain a prediction result corresponding to the sample image.

[0113] The prediction result at least includes: position information of the text area in the sample image.

[0114] S105: Adjust model parameters of the text detection model based on the label information of each sample image and the corresponding prediction result.

[0115] For each sample image, after obtaining the image features of the sample image and the visual cue features of each text area in the sample image, the image features of the sample image and the visual cue features of each text area in the sample image can be respectively input into the text detection model to be trained.

[0116] Among them, optionally, the above-mentioned text detection model can be called a visual-language model, and in the model training process, the input of the visual-language model can include image-shaped data and text-shaped data, and for the embodiment of the present application, the training data of the visual-language model includes image features of sample images, visual cue features of each text area in the sample image, and sample images with label information. Among them, the image features of the sample image and the visual cue features of each text area in the sample image are text-shaped data, and the sample image with label information is image-shaped data. Of course, the above-mentioned text detection model can also be various other models that can meet the text detection requirements of the embodiment of the present application.

[0117] In this way, after receiving the above-mentioned image features and visual cue features, the above-mentioned text detection model can, for each sample image, respectively fuse the image features of the sample image and the visual cue features of each text area in the sample image to obtain a fused feature.

[0118] For example, sample image A includes text areas A1 and A2, where the image feature of sample image A is feature a0, the visual cue feature of text area A1 is feature a1, and the visual cue feature of text area A2 is feature a2. After the above features a0, a1, and a2 are input into the text detection model to be trained, the text detection model can perform a fusion operation on feature a0 and feature a1 to obtain a fused feature M1, and perform a fusion operation on feature a0 and feature a2 to obtain a fused feature M2.

[0119] That is, for each sample image, the number of fused features obtained by the text detection model performing the fusion operation is the same as the number of visual cue features of each text region in the received sample image.

[0120] Furthermore, a position detector for text positioning may be provided in the text detection model to be trained. Then, for each sample image, the text detection model to be trained may perform predictive analysis based on the fusion features obtained by the fusion operation and use the position detector to predict the position information of the text area in the sample image, thereby obtaining a prediction result corresponding to the sample image, and the prediction result at least includes the position information of the text area in the sample image.

[0121] Among them, for each sample image, the position information included in the above prediction result is the prediction value obtained by performing text detection on the sample image during the model training process, and the position information in the label information of the sample image is the true value obtained by performing text detection on the sample image during the model training process, and the training purpose of the text detection model is to make the above prediction value as close to the above true value as possible. Therefore, after obtaining the prediction result corresponding to each sample image, the model parameters of the text detection model can be adjusted based on the label information of each sample image and the corresponding prediction result. In this way, the text detection model after the model parameter adjustment will again obtain the prediction result corresponding to each sample image based on the above fusion result and the position detector, so that the model parameters of the text detection model can be adjusted again based on the label information of each sample image and the corresponding prediction result. The cycle is repeated until the difference between the above prediction value and the above true value for each sample image meets the preset difference requirement, that is, until the difference between the label information of each sample image and the corresponding prediction result meets the preset difference requirement, then the model training can be stopped, and the text detection model after the last model parameter adjustment is the trained text detection model.

[0122] Among them, during the model training process, the difference between the predicted value of the sample data obtained by the model and the true value of the sample data can usually be expressed by a loss function.

[0123] Based on this, optionally, in the above step S104, the loss function of the text detection model to be trained can be pre-constructed, so as to determine whether the above loss function meets the preset requirements based on the difference between the label information of each sample image and the corresponding prediction result. If it does, the current text detection model can be directly determined as the trained text detection model. If it does not, the model parameters of the text detection model can be adjusted, and after the model parameters are adjusted, it is determined again whether the above loss function meets the preset requirements. The cycle is repeated until the loss function meets the preset requirements, and the model training can be stopped. At this time, the text detection model after the last model parameter adjustment is the trained text detection model.

[0124] That is to say, in an embodiment of the present application, when training the text detection model to be trained, the training data used includes: image features of each sample image, visual cue features of each text area in each sample image, and sample images with label information, and, in this specific implementation, the label information includes: location information of the text area in the sample image.

[0125] Optionally, in a specific implementation, the above-mentioned text detection model to be trained may be provided with a character recognizer for text recognition in addition to a position detector for text positioning. Then, when the text detection model to be trained is trained, for each sample image, while the image features of the sample image and the visual cue features of each text area in the sample image are input into the text detection model to be trained, the text cue features of each text area in the sample image are also input into the text detection model to be trained. In this way, during the training process, the text detection model to be trained can perform a fusion operation on the image features of the sample image and the visual cue features and text cue features of each text area in the sample image, respectively, and based on the fusion features obtained by the fusion operation, use the position detector and the character recognizer to perform a predictive analysis to predict the position information of the text area in the sample image.

[0126] Among them, based on the above-mentioned fusion features, the above-mentioned position detector can detect the correlation between the image features of the sample image and the visual prompt features of each text area, thereby predicting the position information of each text area. For example, based on the above-mentioned fusion features, the above-mentioned position detector can detect the position information of the features in the image features of the sample image that are the same or similar to the visual prompt features of each text area, and the detected position information can be used as the predicted position information of each text area. Similarly, the above-mentioned character recognizer can identify the correlation between the image features of the sample image and the text prompt features of each text area, thereby predicting the position information of each text area. For example, based on the above-mentioned fusion features, the above-mentioned character recognizer can identify the position information of the features in the image features of the sample image that are the same or similar to the text prompt features of each text area, and the recognized position information can be used as the predicted position information of each text area. In this way, based on the position information predicted by the position detector and the character recognizer respectively, the position information of the text area in the sample image can be comprehensively determined as the prediction result corresponding to the sample image.

[0127] In addition, in some cases, there may be text areas of multiple preset text types in the image to be detected in the target industrial scene. However, in actual applications, text detection is only performed on text areas of some preset text types, and there is no need to perform text detection on all text areas of the preset text types in the image to be detected.

[0128] For example, there is a text area A belonging to the first category of preset text types, a text area B belonging to the second category of preset text types, and a text area C belonging to the third category of preset text types in the image to be detected. However, in actual applications, only the text areas of the first category of preset text types need to be detected.

[0129] Based on this, optionally, when the text type of the text area that needs to be detected is only: some of the preset text types among the multiple preset text types existing in the image to be detected, then when training the text detection model to be trained, for each sample image in the sample image set, only the visual cue features of the text area of ​​the preset text type that needs to be detected in the sample image can be determined, without determining the visual cue features of the text areas of all the preset text types in the sample image.

[0130] In other words, according to the text detection requirements of the images to be detected in the target industrial scenario, the text areas of the preset text types required in the training process of the text detection model can be determined. This can not only reduce the data processing pressure in the model training process, but also improve the pertinence and detection efficiency of the trained text detection model.

[0131] It can be seen from the above that, when applying the solution provided by the embodiment of the present application, when training a text detection model, the training data used are the image features of the sample image and the visual cue features of each text area in the sample image, wherein the sample image is obtained by image acquisition of a variety of existing scenes, and the label information of each sample image includes at least the position information of the text area in the sample image, and the types of text areas in the multiple sample images include a variety of preset text types. During the training process, the text detection model can predict the position information of the text area in the sample image based on the fusion features of the image features of the sample image and the visual cue features of the text area, and then, the model parameters of the text detection model can be adjusted based on the label information of the sample image and the corresponding prediction results.

[0132] Based on this, since in the model training process, the fusion features learned by the text detection model come from multiple sample images collected under various existing scenes, and multiple text areas belonging to each preset text type in the above multiple sample images, the text detection model can learn the correlation characteristics between the text areas of the preset text types under the scene and the scene and the characteristics of the text area itself during training, so that the text detection model obtained by training has a higher generalization ability, and therefore, the trained text detection model can detect the text areas of each preset text type in the image outside the sample image set. That is, the trained text detection model can detect the text areas of each preset text type in the scene image of the scene to be detected without relying on the scene image of the scene to be detected as a sample image. In this way, the scheme provided by the application embodiment can be realized without relying on the scene image of the industrial scene to be detected as a sample image, and the text detection model suitable for the industrial scene to be detected can be trained.

[0133] In addition, since the sample images in the sample image set can come from a variety of existing scenes, rather than being limited to industrial scenes, a large number of sample images can be obtained for training the text detection model. Based on this, using a large number of sample images to train the text detection model can avoid overfitting of the trained text detection model, reduce the missed detection rate of the trained text detection model, and improve the detection efficiency of the trained text detection model.

[0134] Optionally, in a specific implementation, such as Figure 3 As shown, the training method of a text detection model based on visual cues provided in an embodiment of the present application may also include the following step S106:

[0135] S106: Acquire text prompt information of each text region in each sample image, and determine a text prompt feature of the text prompt information of each text region as the text prompt feature of the text region.

[0136] Accordingly, in this specific implementation, the above step S104, for each sample image, inputs the image features of the sample image and the visual cue features of each text area in the sample image into the text detection model to be trained, so that the text detection model performs a fusion operation on the image features of the sample image and the visual cue features of each text area in the sample image, and based on the fusion features obtained by the fusion operation, performs a prediction analysis using a position detector to obtain a prediction result corresponding to the sample image, which may include the following step S1041:

[0137] S1041: for each sample image, inputting the image features of the sample image, the visual cue features of each text region in the sample image, and the text cue features of each text region in the sample image into the text detection model to be trained, so that the text detection model performs a multimodal feature fusion operation on the image features of the sample image and the target cue features of each text region in the sample image, and performs prediction analysis using a position detector based on the fusion features obtained by the fusion operation to obtain a prediction result corresponding to the sample image;

[0138] The target prompt feature of each text region includes: a visual prompt feature and a text prompt feature of the text region.

[0139] In this specific implementation, in order to further improve the generalization ability of the trained text detection model and to improve the applicability and detection accuracy of the trained text detection model, the character content in the text area of ​​the sample image can also be used for model training.

[0140] For each sample image, the text prompt information of each text area in the sample image can be first obtained, and then the text prompt features of the text prompt information of each text area can be determined as the text prompt features of the text area. For example, a text prompt feature extractor can be used, such as a language model based on a Transformer network structure, to extract the language features of each text area as the text prompt features of the text area.

[0141] That is to say, in this specific implementation, when training the text detection model to be trained, the training data used includes: image features of each sample image, visual cue features of each text area in each sample image, text cue features of each text area in each sample image, and sample images with label information, and, in this specific implementation, the label information includes: location information of the text area in the sample image.

[0142] Optionally, in a specific implementation, in the above step S106 , the step of obtaining text prompt information of each text area in each sample image may include the following step 1061 , or step 1062 .

[0143] Step 1061: Obtain the character content of each text area in each sample image as text prompt information of each text area;

[0144] In this specific implementation, for each sample image, the character content of each text area in the sample image can be directly obtained, so that the character content is used as the text prompt information of the text area.

[0145] For example, taking the text area in the sample image used to indicate the production date as an example, if the character content of the text area in the sample image is "2023-01-01", used to indicate that the production date of the object captured by the sample image is "2023-01-01", then the text line "2023-01-01" in the text area can be directly used as the text prompt information of the text area.

[0146] Step 1062: Obtain the expression of the character content of each text area in each sample image as the text prompt information of each text area.

[0147] In this specific implementation, for each sample image and each text area in the sample image, the character content of the text area can be first obtained, and then, based on the specific content of the character content, an expression for characterizing the writing features of the character content can be determined. The expression is the text prompt information of the text area.

[0148] For example, taking the text area in the sample image used to represent the production date as an example, if the character content of the text area in the sample image is "2023-01-01", used to represent that the production date of the object captured by the sample image is "2023-01-01", then it can be determined that the regular expression of the text line "2023-01-01" in the text area is ".*\d{4}-\d{2}-\d{2}.*", and further, the above ".*\d{4}-\d{2}-\d{2}.*" can be used as the text prompt information of the text area.

[0149] For another example, taking the text area in the sample image used to indicate the production date as an example, if the character content of the text area in the sample image is "2023-01-01", used to indicate that the production date of the object captured by the sample image is "2023-01-01", then the text line type of the text area is "a text line mixed with symbols and numbers", that is, the text type of the text area is "a text line mixed with symbols and numbers", then the above text type "a text line mixed with symbols and numbers" can also be used as the text prompt information of the text area.

[0150] Among them, when the expression of the character content of each text area in each sample image is used as the text prompt information of each text area, compared with the related technology, the character content of the text area in the sample image is used as a label to use fixed prompt information for model training. In this specific implementation method, the expression of the character content of the text area is used as the text prompt information, and the model training can be carried out through dynamic prompt information, thereby reducing the possibility of missed detection of the trained text detection model, thereby further improving the generalization ability and detection accuracy of the trained text detection model.

[0151] That is to say, in the existing deep learning field, target text line positioning and recognition methods are mostly based on the precise character content of the target text line or text questions and answers to detect and recognize all positions and contents of the character appearing in the image, while in the industry, the text lines required for actual scenarios are less than the number of text lines in the current sample, and the texts with similar content are diverse. Therefore, only using fixed prompt characters makes it easy for deep learning methods to miss detections. Therefore, in the embodiments of the present application, the text prompt features and visual prompt features of the text area are extremely important. By using the expression of the character content of the text area as the text prompt information, dynamic text prompt information can be achieved through the text area, reducing the possibility of missed detection by the trained text detection model.

[0152] That is, for each sample image, there are image features of the sample image, visual cue features of each text area in the sample image, and text cue features of each text area in the sample image, and the visual cue features and text cue features of the same text area can be collectively regarded as the target cue features of the text area.

[0153] In this way, for each sample image, the image features of the sample image, the visual cue features of each text area in the sample image, and the text cue features of each text area in the sample image can be respectively input into the text detection model to be trained. In this way, after receiving the above-mentioned image features, visual cue features, and text cue features, the above-mentioned text detection model can perform a multimodal fusion operation on the image features of the sample image, the visual cue features of the text area, and the text cue features of the text area for each text area in the sample image to obtain a fused feature. That is, for each sample image, the text detection model to be trained can perform a multimodal feature fusion operation on the image features of the sample image and the target cue features of each text area in the sample image.

[0154] Among them, the so-called multimodality means: extracting image features and text line sequence features respectively through two different models or neural networks, and fully combining the text context semantic information to promote the model or neural network to learn in a direction that is beneficial to the current text information.

[0155] Among them, optionally, the above-mentioned text detection model can perform a multimodal feature fusion operation on the image features of the sample image and the target prompt features of each text area in the sample image by splicing the image features of the sample image and the target prompt features of each text area in the sample image, that is, for each text area in the sample image, the above-mentioned text detection model can splice the image features of the sample image, the visual prompt features of the text area, and the text prompt features of the text area. In this way, the result obtained by splicing is the obtained fusion feature.

[0156] Of course, the above-mentioned text detection model can also perform the above-mentioned multimodal feature fusion in other ways, for example, module fusion of various features, etc., and the embodiments of the present application do not make specific limitations on this.

[0157] For example, sample image A includes text areas A1 and A2, wherein the image feature of sample image A is feature a0, the visual cue feature and text cue feature of text area A1 are features a1 and b1 respectively, and the visual cue feature and text cue feature of text area A2 are features a2 and b2 respectively. After the above features a0, a1, a2, b1 and b2 are input into the text detection model to be trained, the text detection model can perform a multimodal feature fusion operation on features a0, a1 and b1 to obtain fused feature m1, and perform a multimodal feature fusion operation on features a0, a2 and b2 to obtain fused feature m2.

[0158] With the above Figure 1Similar to the specific implementation shown, in this specific implementation, for each sample image, the number of fused features obtained by the text detection model performing the fusion operation is the same as the number of visual cue features of each text area in the received sample image.

[0159] Among them, optional, in such Figure 2 In the specific implementation shown, the above step S106 includes the following step 1061:

[0160] Step 1061: Acquire text prompt information of each text area belonging to the target text type in each sample image, and determine text prompt features of the text prompt information of each text area belonging to the target text type as text prompt features of the text area belonging to the target text type.

[0161] That is to say, in this specific implementation, for each sample image, the visual cue features and text cue features input into the text detection model to be trained are: the visual cue features and text cue features of each text area belonging to the target text type in the sample image.

[0162] Furthermore, after obtaining the above fusion features, for each sample image, the text detection model to be trained can use the position detector to perform prediction analysis based on the fusion features obtained by the fusion operation to predict the position information of the text area in the sample image, thereby obtaining the prediction result corresponding to the sample image, and the prediction result at least includes the position information of the text area in the sample image. Furthermore, the model parameters of the text detection model can be adjusted based on the label information of each sample image and the corresponding prediction result.

[0163] In many cases, the purpose of text detection on an image is to perform text recognition on the image, that is, to recognize the character content of the text area in the image. Therefore, when training a text detection model, the trained text detection model can recognize the character content of the text to be detected in the image to be detected and directly output the above character content. In order to train a text detection model capable of performing the above text recognition, the label information of each sample image in the above sample image set must also include the character content of the text area in the sample image, and in the model training process, the character content of the text area in the sample image will be used as the true value of the text detection model to be trained to detect the sample image and participate in the training.

[0164] Based on this, optionally, in a specific implementation, the label information of each sample image further includes: character content of a text area in the sample image;

[0165] Accordingly, in this specific implementation, if Figure 4As shown, the above step S1041, for each sample image, respectively inputs the image features of the sample image, the visual cue features of each text area in the sample image, and the text cue features of each text area in the sample image into the text detection model to be trained, so that the text detection model performs a multimodal feature fusion operation on the image features of the sample image and the target cue features of each text area in the sample image, and based on the fusion features obtained by the fusion operation, performs a prediction analysis using a position detector to obtain a prediction result corresponding to the sample image, which may include the following step S1041A:

[0166] S1041A: For each sample image, the image features of the sample image, the visual prompt features of each text area in the sample image, and the text prompt features of each text area in the sample image are respectively input into the text detection model to be trained, so that the text detection model performs a multimodal feature fusion operation on the image features of the sample image and the target prompt features of each text area in the sample image, and based on the fused features obtained by the fusion operation, uses a position detector and a character recognizer to perform predictive analysis to obtain a prediction result corresponding to the sample image.

[0167] The prediction result also includes: the character content of the text area in the sample image.

[0168] In this specific implementation, the text detection model to be trained can be provided with a character recognizer for text recognition on the basis of a position detector for text positioning. Then, for each sample image, the text detection model to be trained can perform predictive analysis based on the fusion features obtained by the fusion operation, using the position detector and the character recognizer to predict the position information and character content of the text area in the sample image, and obtain the prediction result corresponding to the sample image, which includes the position information and character content of the text area in the sample image.

[0169] Among them, the text detection model to be trained can first perform predictive analysis based on the fusion features obtained by the fusion operation, using a position detector to predict the position information of the text area in the sample image, and then, based on the fusion features obtained by the fusion operation, use a character recognizer to perform text recognition on the text area corresponding to the above-mentioned predicted position information to obtain the character content of the text area in the sample image.

[0170] For each sample image, the position information and character content included in the above prediction results are the predicted values ​​obtained by performing text detection on the sample image during the model training process, and the position information and character content in the label information of the sample image are the true values ​​obtained by performing text detection on the sample image during the model training process. The training purpose of the text detection model is to make the above prediction values ​​as close to the above true values ​​as possible. Therefore, after obtaining the prediction results corresponding to each sample image, the model parameters of the text detection model can be adjusted based on the label information of each sample image and the corresponding prediction results. In this way, the text detection model after the model parameters are adjusted will again obtain the prediction results corresponding to each sample image based on the above fusion results, using the position detector and the character recognizer, so that the model parameters of the text detection model can be adjusted again based on the label information of each sample image and the corresponding prediction results. The process is repeated repeatedly until the difference between the predicted value and the true value of each sample image meets the preset difference requirement, that is, until the difference between the label information of each sample image and the corresponding prediction result meets the preset difference requirement. Then, the model training can be stopped. At this time, the text detection model after the last model parameter adjustment is the trained text detection model.

[0171] That is to say, in this specific implementation, when training the text detection model to be trained, the training data used includes: image features of each sample image, visual prompt features of each text area in each sample image, text prompt features of each text area in each sample image, and sample images with label information, and, in this specific implementation, the label information includes: position information and character content of the text area in the sample image.

[0172] In order to understand the training method of the text detection model provided in the embodiment of the present application, Figure 5 The above-mentioned embodiments provided in this application Figure 4 The model reasoning flow chart of the specific implementation method is shown in FIG. Figure 5 As shown:

[0173] 1. The region map 501 is: the text region in the sample image 500, i.e., the text region in the sample image;

[0174] 2. The target character prompt information 502 is: text prompt information of the character content in the area diagram 501 represented by a regular expression;

[0175] 3. The region feature extractor 503 is used to extract visual cue features of the region map 501, and input the extracted visual cue features to the encoder 505 based on the map or text features; that is, the region feature extractor 503 is the visual cue described above.

[0176] 4. The text feature extractor 504 is used to extract the text prompt features of the target character prompt information 502, and input the extracted text prompt features to the encoder 505 based on the graph or text feature; that is, the text feature extractor 504 is the text prompt feature extractor described above.

[0177] 5. The encoder 505 based on the image or text feature is used to extract the image features of the sample image 500, and perform a multimodal feature fusion operation on the extracted image features, the received visual prompt features and the text prompt features, and send the obtained fusion features to the position detector 506 and the character recognizer 507; that is, the encoder 505 based on the image or text feature is the encoder described above;

[0178] 6. The position detector 506 is used to predict the position information of the region map 501 based on the received fusion features;

[0179] 7. The character recognizer 507 is used to recognize the character content of the area map 501 based on the received fusion features;

[0180] 8. So far, through the prediction of the position detector 506 and the character recognizer 507, the final prediction result 508 can be obtained, that is, the character content of the text area to be detected in the sample image 500 is detected.

[0181] Corresponding to the above Figure 4 The specific implementation method shown and Figure 5 The model reasoning flowchart shown, Figure 6 The above-mentioned embodiments provided in this application Figure 4 The structural diagram of the text detection model of the specific implementation method is shown in FIG. Figure 6 As shown:

[0182] The above text detection model may include: an encoder 601 , a position detector 602 , and a character recognizer 603 .

[0183] When the visual prompt feature of the text area in the sample image is extracted by the visual prompter, the extracted visual prompt feature can be sent to the encoder 601; and when the text prompt feature extractor is used to extract the text prompt feature of the text area in the sample image, the extracted text prompt feature can also be sent to the encoder 601;

[0184] The encoder 601 is used to extract image features of the sample image, and perform a multimodal feature fusion operation on the image features and the received visual prompt features and text prompt features to obtain fused features;

[0185] A position detector 602, used for predicting and analyzing the position information of the text area in the sample image based on the fusion features;

[0186] The character recognizer 603 is used to perform text recognition on the text area corresponding to the position information predicted by the position detector 602 based on the fusion features, and obtain the character content of the text area of ​​the sample image.

[0187] As can be seen from the above, in the prior art, the existing single-stage method for locating the target text line uses a multi-scale feature encoder and decoder, the network structure is cumbersome and the difficulty of industrial deployment is very high. The multi-stage method uses a parallel network to extract text features and features to be detected for feature similarity learning, and does not give full play to the role of prompt features. In addition, the method of single-character positioning based on sliding windows is difficult in data annotation, and the effect is poor for characters with rotation directions. Compared with the above-mentioned existing single-stage method, in the embodiment of the present application, an encoder can be used to realize the multimodal fusion of image features, visual prompt features and text prompt features of the text area, without setting a multi-scale feature encoder and decoder, simplifying the structure of the text detection model and reducing the difficulty of industrial deployment. At the same time, through the multimodal fusion operation, the fused features are used to predict the position of the text area and recognize the characters, which can give full play to the role of various prompt features, and the character content of the text area is used as a whole for text feature extraction, and there is no need to perform single-character positioning on the characters in the text area, thereby greatly reducing the difficulty of data annotation, and even for characters with rotation directions, a good detection effect can be achieved.

[0188] In response to the above-mentioned embodiment of the present application provides a training method for a text detection model based on visual cues, the embodiment of the present application also provides a text detection method based on visual cues.

[0189] Figure 7 A flowchart of a text detection method based on visual cues provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, the method may include the following steps S701-S702:

[0190] S701: Acquire an image to be detected in a target industrial scene.

[0191] The text type of the text area in the image to be detected is at least one of a plurality of preset text types.

[0192] S702: Input the image to be detected into a pre-trained text detection model to obtain a detection result.

[0193] Among them, the text detection model is trained based on the training method of any text detection model provided in the above-mentioned embodiments of the present application.

[0194] In an embodiment of the present application, an industrial scene that requires text detection can be used as a target industrial scene. After the text detection model is trained using any of the text detection model training methods provided in the above-mentioned embodiments of the present application, the image to be detected in the target industrial scene can be first obtained.

[0195] Since the text area used to extract visual cue features when training the above text inspection model is a text area in the sample image whose text type is a preset text type, the trained text detection model can detect the text area of ​​the preset text type in the scene image. Based on this, in order to realize text detection of the image to be detected in the target industrial scene, the text type of the text area in the above image to be detected is at least one of the multiple preset text types in the above model training method.

[0196] In this way, the image to be detected can be input into the pre-trained text detection model, and then the above text detection model can perform text detection on the image to be detected and output the detection result of the image to be detected.

[0197] Among them, optionally, if the label information of the sample image used to train the above-mentioned text detection model includes the position information of the text area in the sample image, and during the training process, the prediction result obtained by the text detection model through prediction analysis includes the position information of the text area in the sample image, then, when performing text detection, the above-mentioned detection result is: the position information of the text area in the image to be detected. And, at this time, the above-mentioned text detection model can be provided with a position detector for text positioning, and can also be provided with a position detector for text positioning and a character recognizer for text recognition at the same time.

[0198] Optionally, if the label information of the sample image used to train the above-mentioned text detection model includes the position information and character content of the text area in the sample image, and, during the training process, the prediction result obtained by the text detection model using the position detector for predictive analysis includes the position information and character content of the text area in the sample image, then, when performing text detection, the above-mentioned text detection model can directly output the character content of the text area in the image to be detected, that is, the above-mentioned detection result is: the character content of the text area in the image to be detected. And, at this time, the above-mentioned text detection model can be provided with a position detector for text positioning, and can also be provided with a position detector for text positioning and a character recognizer for text recognition at the same time.

[0199] As can be seen from the above, by applying the solution provided in the embodiment of the present application, it is possible to train a text detection model suitable for the industrial scene to be detected without relying on the scene image of the industrial scene to be detected as a sample image. In addition, it is possible to avoid overfitting of the trained text detection model, reduce the missed detection rate of the trained text detection model, and improve the detection efficiency of the trained text detection model.

[0200] Corresponding to a method for training a text detection model based on visual cues provided in an embodiment of the present application, an embodiment of the present application also provides a device for training a text detection model based on visual cues.

[0201] Figure 8 A training device for a text detection model based on visual cues is provided in an embodiment of the present application, such as Figure 8 As shown, the device may include the following modules:

[0202] The sample image acquisition module 810 is used to acquire a sample image set; wherein the sample image set includes a plurality of sample images obtained by collecting images of a plurality of existing scenes, the types of text areas in the plurality of sample images include a plurality of preset text types, and each sample image has: label information at least including position information of the text area in the sample image;

[0203] The first feature extraction module 820 is used to extract the image features of each sample image and determine the visual cue features of each text region in each sample image; wherein the visual cue features of each text region are used to characterize: the attributes of the region under the text type to which the text region belongs;

[0204] The feature fusion module 830 is used to input the image features of the sample image and the visual cue features of each text area in the sample image to the text detection model to be trained for each sample image, so that the text detection model performs a fusion operation on the image features of the sample image and the visual cue features of each text area in the sample image, and performs a prediction analysis based on the fusion features obtained by the fusion operation using a position detector to obtain a prediction result corresponding to the sample image; wherein the prediction result at least includes: the position information of the text area in the sample image;

[0205] The parameter adjustment module 840 is used to adjust the model parameters of the text detection model based on the label information of each sample image and the corresponding prediction result.

[0206] It can be seen from the above that, when applying the solution provided by the embodiment of the present application, when training a text detection model, the training data used are the image features of the sample image and the visual cue features of each text area in the sample image, wherein the sample image is obtained by image acquisition of a variety of existing scenes, and the label information of each sample image includes at least the position information of the text area in the sample image, and the types of text areas in the multiple sample images include a variety of preset text types. During the training process, the text detection model can predict the position information of the text area in the sample image based on the fusion features of the image features of the sample image and the visual cue features of the text area, and then, the model parameters of the text detection model can be adjusted based on the label information of the sample image and the corresponding prediction results.

[0207] Based on this, since in the model training process, the fusion features learned by the text detection model come from multiple sample images collected under various existing scenes, and multiple text areas belonging to each preset text type in the above multiple sample images, the text detection model can learn the correlation characteristics between the text areas of the preset text types under the scene and the scene and the characteristics of the text area itself during training, so that the text detection model obtained by training has a higher generalization ability, and therefore, the trained text detection model can detect the text areas of each preset text type in the image outside the sample image set. That is, the trained text detection model can detect the text areas of each preset text type in the scene image of the scene to be detected without relying on the scene image of the scene to be detected as a sample image. In this way, the scheme provided by the application embodiment can be realized without relying on the scene image of the industrial scene to be detected as a sample image, and the text detection model suitable for the industrial scene to be detected can be trained.

[0208] In addition, since the sample images in the sample image set can come from a variety of existing scenes, rather than being limited to industrial scenes, a large number of sample images can be obtained for training the text detection model. Based on this, using a large number of sample images to train the text detection model can avoid overfitting of the trained text detection model, reduce the missed detection rate of the trained text detection model, and improve the detection efficiency of the trained text detection model.

[0209] Optionally, in a specific implementation, the device further includes:

[0210] A second feature extraction module is used for obtaining text prompt information of each text area in each sample image and determining text prompt features of the text prompt information of each text area as the text prompt features of the text area before inputting the image features of the sample image and the visual prompt features of each text area in the sample image into the text detection model to be trained for each sample image;

[0211] The feature fusion module 830 is specifically used to input the image features of each sample image, the visual prompt features of each text area in the sample image, and the text prompt features of each text area in the sample image into the text detection model to be trained, so that the text detection model performs a multimodal feature fusion operation on the image features of the sample image and the target prompt features of each text area in the sample image; wherein the target prompt features of each text area include: the visual prompt features and the text prompt features of the text area.

[0212] Optionally, in a specific implementation, the label information of each sample image further includes: character content of a text area in the sample image;

[0213] The feature fusion module 830 is specifically used to input the image features of each sample image, the visual prompt features of each text area in the sample image, and the text prompt features of each text area in the sample image into the text detection model to be trained, so that the text detection model performs a multimodal feature fusion operation on the image features of the sample image and the target prompt features of each text area in the sample image, and based on the fusion features obtained by the fusion operation, uses a position detector and a character recognizer to perform prediction analysis to obtain a prediction result corresponding to the sample image; wherein the prediction result also includes: the character content of the text area in the sample image.

[0214] Optionally, in a specific implementation, the second feature extraction module is specifically used to:

[0215] Obtaining character content of each text area in each sample image as text prompt information of each text area;

[0216] or,

[0217] An expression of the character content of each text area in each sample image is obtained as text prompt information of each text area.

[0218] Optionally, in a specific implementation, the device further includes:

[0219] The image acquisition module is used to acquire scene images of various existing industrial scenes as multiple sample images.

[0220] Optionally, in a specific implementation, the first feature extraction module 820 is specifically used to:

[0221] A target text type related to a target industrial scene is determined, and visual cue features of each text region belonging to the target text type in each sample image are determined.

[0222] Corresponding to the text detection method based on visual cues provided in the above-mentioned embodiment of the present application, the embodiment of the present application also provides a text detection device based on visual cues.

[0223] Fig. 9 A structural diagram of a text detection device based on visual cues provided in an embodiment of the present application is shown in FIG. Fig. 9 As shown, the device may include the following modules:

[0224] The image acquisition module 910 is used to acquire an image to be detected in a target industrial scene; wherein the text type of the text area in the image to be detected is: at least one of a plurality of preset text types;

[0225] The result detection module 920 is used to input the image to be detected into a pre-trained text detection model to obtain a detection result; wherein the text detection model is trained based on the training method of any text detection model provided in the embodiments of the present application.

[0226] As can be seen from the above, by applying the solution provided in the embodiment of the present application, it is possible to train a text detection model suitable for the industrial scene to be detected without relying on the scene image of the industrial scene to be detected as a sample image. In addition, it is possible to avoid overfitting of the trained text detection model, reduce the missed detection rate of the trained text detection model, and improve the detection efficiency of the trained text detection model.

[0227] The present application also provides an electronic device, such as Fig.10 As shown, including:

[0228] Memory 1001, used for storing computer programs;

[0229] The processor 1002 is used to implement the steps of the training method of any text detection model provided in the embodiments of the present application, and / or implement the steps of any text detection method provided in the embodiments of the present application when executing the program stored in the memory 1501.

[0230] Furthermore, the electronic device may further include a communication bus and / or a communication interface, and the processor 1002, the communication interface, and the memory 1001 communicate with each other via the communication bus.

[0231] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0232] The communication interface is used for communication between the above electronic device and other devices.

[0233] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0234] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0235] Optional, Fig.11 This is a structural diagram of an electronic device. The electronic device 1100 may include a processor 1110, a display screen 1120, a camera 1130, an internal memory 1140, a Universal Serial Bus (USB) interface 1150, a charging management module 1160, a battery management module 1161, a battery 1162, a sensor module 170, a mobile communication module 180, a wireless communication module 190, an antenna 1 and an antenna 2, etc.

[0236] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 1100. In other embodiments of the present application, the electronic device 1100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0237] The processor 1110 may include one or more processing units, for example: the processor 1110 may include a central processing unit (CPU), an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processor (NPU), etc. Among them, different processing units may be independent components or integrated into one or more processors. In some embodiments, the electronic device 1100 may also include one or more processors 1110. In some other embodiments, a memory may also be provided in the processor 1110 for storing instructions and data. Exemplarily, the memory in the processor 1110 may be a cache memory. The memory may store instructions or data that the processor 1110 has just used or circulated. If the processor 1110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated accesses, reduces the waiting time of the processor 1110, and thus improves the efficiency of the electronic device 1100 in processing data or executing instructions.

[0238] In some embodiments, the processor 1110 may include one or more interfaces. The interface may include an Inter-Integrated Circuit (I2C) interface, an Inter-Integrated Circuit Sound (I2S) interface, a Pulse Code Modulation (PCM) interface, a Universal Asynchronous Receiver / Transmitter (UART) interface, a Mobile Industry Processor Interface (MIPI), a General-Purpose Input / Output (GPIO) interface, a SIM card interface and / or a USB interface, etc. Among them, the USB interface 1150 is an interface that complies with the USB standard specification, and may specifically be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 1150 may be used to connect a charger to charge the electronic device 1100, and may also be used to transmit data between the electronic device 1100 and a peripheral device. The USB interface 1150 may also be used to connect headphones to play audio through the headphones.

[0239] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is for illustrative purposes only and does not constitute a structural limitation on the electronic device 1100. In other embodiments of the present application, the electronic device 1100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0240] The wireless communication function of the terminal 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 180, the wireless communication module 190, the modem processor and the baseband processor.

[0241] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in terminal 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0242] The electronic device 1100 implements display functions through a GPU, a display screen 1120, and an application processor, etc. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0243] The display screen 1120 is used to display images, videos, etc. The display screen 1120 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 1100 may include one or more display screens 1120.

[0244] The electronic device 1100 can implement a shooting function through an ISP, a camera 1130, a video codec, a GPU, a display screen 1120, and an application processor, wherein the camera 1130 includes a front camera and a rear camera.

[0245] The camera 1130 is used to take photos or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element may be a charge coupled device (CCD). The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard red green blue (RGB), YUV or other format. In some embodiments, the electronic device 1100 may include 1 or N cameras 1130, where N is a positive integer greater than 1.

[0246] The internal memory 1140 may be used to store one or more computer programs, which include instructions. The processor 1110 may enable the electronic device 1100 to perform the model training method and text detection method provided in the embodiment of the present application, as well as various applications and data processing, by running the above instructions stored in the internal memory 1140. In addition, the internal memory 1140 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more disk storage components, a flash memory component, a universal flash memory (Universal Flash Storage, UFS), etc. In some embodiments, the processor 1110 may enable the electronic device 1100 to perform the model training method and text detection method provided in the embodiment of the present application, as well as other applications and data processing, by running instructions stored in the internal memory 140, and / or instructions stored in a memory provided in the processor 1110.

[0247] The internal memory 1140 can be used to store relevant programs of the method provided in the embodiment of the present application, and the processor 110 can be used to call relevant programs of the model training method and the text detection method stored in the internal memory 140 when displaying information, and execute the model training method and the text detection method of the embodiment of the present application.

[0248] The sensor module 1170 may include a pressure sensor, an infrared emitter, an infrared light detector, a touch sensor, an ambient light sensor, and the like.

[0249] In another embodiment provided in the present application, a computer-readable storage medium is also provided, which stores a computer program. When the computer program is executed by a processor, it implements the steps of the training method of any text detection model provided in the embodiments of the present application, and / or implements the steps of any text detection method provided in the embodiments of the present application.

[0250] In another embodiment provided in the present application, a computer program product comprising instructions is also provided. When the computer program product is run on a computer, the computer executes the steps of the training method of any text detection model provided in the embodiments of the present application, and / or implements the steps of any text detection method provided in the embodiments of the present application.

[0251] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk (SSD), etc.

[0252] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0253] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for device embodiments, electronic device embodiments, computer-readable storage medium embodiments, and computer program product embodiments, since they are basically similar to method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The above is only a preferred embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application are included in the scope of protection of the present application.

Claims

1. A method for training a text detection model based on visual cues, characterized in that: The method comprises: Acquire a sample image set; wherein the sample image set includes a plurality of sample images obtained by collecting images of a plurality of existing scenes, the types of text areas in the plurality of sample images include a plurality of preset text types, and each sample image has: label information at least including position information of the text area in the sample image; Extracting image features of each sample image, and determining visual cue features of each text region in each sample image; wherein the visual cue features of each text region are used to characterize: the attributes of the region under the text type to which the text region belongs; For each sample image, the image features of the sample image and the visual cue features of each text area in the sample image are respectively input into the text detection model to be trained, so that the text detection model performs a fusion operation on the image features of the sample image and the visual cue features of each text area in the sample image, and based on the fusion features obtained by the fusion operation, a position detector is used to perform a prediction analysis to obtain a prediction result corresponding to the sample image; wherein the prediction result at least includes: the position information of the text area in the sample image; Based on the label information of each sample image and the corresponding prediction result, the model parameters of the text detection model are adjusted.

2. The method according to claim 1, characterized in that Before inputting the image features of each sample image and the visual cue features of each text area in the sample image into the text detection model to be trained, the method further includes: Acquire text prompt information of each text region in each sample image, and determine a text prompt feature of the text prompt information of each text region as the text prompt feature of the text region; For each sample image, the image features of the sample image and the visual cue features of each text area in the sample image are respectively input into the text detection model to be trained, so that the text detection model performs a fusion operation on the image features of the sample image and the visual cue features of each text area in the sample image, including: For each sample image, the image features of the sample image, the visual prompt features of each text area in the sample image, and the text prompt features of each text area in the sample image are respectively input into the text detection model to be trained, so that the text detection model performs a multimodal feature fusion operation on the image features of the sample image and the target prompt features of each text area in the sample image; wherein the target prompt features of each text area include: the visual prompt features and the text prompt features of the text area.

3. The method according to claim 2, characterized in that The label information of each sample image also includes: the character content of the text area in the sample image; The text detection model performs prediction analysis based on the fusion features obtained by the fusion operation using the position detector to obtain a prediction result corresponding to the sample image, including: The text detection model performs prediction analysis based on the fusion features obtained by the fusion operation using a position detector and a character recognizer to obtain a prediction result corresponding to the sample image; wherein the prediction result also includes: the character content of the text area in the sample image.

4. The method according to claim 2, characterized in that: The obtaining of text prompt information of each text area in each sample image includes: Obtaining character content of each text area in each sample image as text prompt information of each text area; or, An expression of the character content of each text area in each sample image is obtained as text prompt information of each text area.

5. The method according to any one of claims 1 to 4, characterized in that: The acquisition method of the multiple sample images includes: Scene images of various existing industrial scenes are collected as multiple sample images.

6. The method according to any one of claims 1 to 4, characterized in that: The step of determining the visual cue features of each text region in each sample image includes: A target text type related to a target industrial scene is determined, and visual cue features of each text region belonging to the target text type in each sample image are determined.

7. A text detection method based on visual cues, characterized in that: The method comprises: Acquire an image to be detected in a target industrial scene; wherein the text type of the text area in the image to be detected is: at least one of a plurality of preset text types; The image to be detected is input into a pre-trained text detection model to obtain a detection result; wherein the text detection model is trained based on the method described in any one of claims 1-6.

8. A training device for a text detection model, characterized in that: The device comprises: A sample image acquisition module, used to acquire a sample image set; wherein the sample image set includes a plurality of sample images obtained by collecting images of a plurality of existing scenes, the types of text areas in the plurality of sample images include a plurality of preset text types, and each sample image has: label information at least including position information of the text area in the sample image; The first feature extraction module is used to extract the image features of each sample image and determine the visual cue features of each text area in each sample image; wherein the visual cue features of each text area are used to characterize: the attributes of the area under the text type to which the text area belongs; A feature fusion module is used to input the image features of each sample image and the visual cue features of each text area in the sample image into the text detection model to be trained, so that the text detection model performs a fusion operation on the image features of the sample image and the visual cue features of each text area in the sample image, and performs a prediction analysis based on the fusion features obtained by the fusion operation using a position detector to obtain a prediction result corresponding to the sample image; wherein the prediction result at least includes: the position information of the text area in the sample image; The parameter adjustment module is used to adjust the model parameters of the text detection model based on the label information of each sample image and the corresponding prediction result.

9. The device according to claim 8, characterized in that The device also includes: A second feature extraction module is used for obtaining text prompt information of each text area in each sample image and determining text prompt features of the text prompt information of each text area as the text prompt features of the text area before inputting the image features of the sample image and the visual prompt features of each text area in the sample image into the text detection model to be trained for each sample image; The feature fusion module is specifically used to input the image features of the sample image, the visual prompt features of each text area in the sample image, and the text prompt features of each text area in the sample image into the text detection model to be trained for each sample image, so that the text detection model performs a multimodal feature fusion operation on the image features of the sample image and the target prompt features of each text area in the sample image; wherein the target prompt features of each text area include: the visual prompt features and the text prompt features of the text area; and / or, The label information of each sample image also includes: the character content of the text area in the sample image; The feature fusion module is specifically used to input the image features of the sample image, the visual prompt features of each text area in the sample image, and the text prompt features of each text area in the sample image into the text detection model to be trained for each sample image, so that the text detection model performs a multimodal feature fusion operation on the image features of the sample image and the target prompt features of each text area in the sample image, and performs prediction analysis based on the fusion features obtained by the fusion operation using a position detector and a character recognizer to obtain a prediction result corresponding to the sample image; wherein the prediction result also includes: the character content of the text area in the sample image; and / or, The second feature extraction module is specifically used for: Obtaining character content of each text area in each sample image as text prompt information of each text area; or, Obtaining the expression of the character content of each text area in each sample image as the text prompt information of each text area; and / or, The device also includes: An image acquisition module, used for acquiring scene images of various existing industrial scenes as multiple sample images; and / or, The first feature extraction module is specifically used for: A target text type related to a target industrial scene is determined, and visual cue features of each text region belonging to the target text type in each sample image are determined.

10. A text detection device based on visual cues, characterized in that: The device comprises: An image acquisition module is used to acquire an image to be detected in a target industrial scene; wherein the text type of the text area in the image to be detected is: at least one of a plurality of preset text types; A result detection module is used to input the image to be detected into a pre-trained text detection model to obtain a detection result; wherein the text detection model is trained based on the method described in any one of claims 1-6.

11. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, for implementing any one of the methods described in claims 1 to 6 and / or implementing the method described in claim 7 when executing a program stored in a memory.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of claims 1 to 6 is implemented, or the method described in claim 7 is implemented.