A method for extracting structured information from images

Abstract

The present invention discloses a method, device, electronic device and storage medium for extracting structured information of an image in the field of image recognition and processing technology, aiming to solve the technical problem of large image recognition errors. It includes: selecting an original image, the original image having a target object and background information; segmenting the target object and background information in the original image according to a pre-constructed semantic segmentation network model to extract the target object; based on the target object, according to a pre-constructed target detection model, identifying the target object and obtaining the target features in the target object; using a histogram to solve the connected domain method to analyze the target features and obtain the feature information in the target features. The present invention can lock the target features, thereby reducing the interference of environmental factors and reducing the errors in target feature recognition.

Description

Technical Field

[0001] The invention relates to a method for extracting structured information of an image, and belongs to the technical field of image recognition and processing. Background Art

[0002] With the continuous development of the security field, image and video data has become increasingly large, but lacks structured description information, making it difficult for information retrieval systems to perform effective retrieval. When reviewing, users can generally only search by time period. Only by structuring images and videos can users quickly and efficiently find content of interest. Image structuring is the process of semanticizing image content. It processes images intelligently and translates them into a language that computers or humans can understand. It uses methods such as image segmentation, object recognition, and feature extraction to obtain text information. Image structuring can be for a single image or a video clip composed of multiple images. Therefore, in the security field, the means of structuring video images of pedestrians, vehicles, etc. are indispensable.

[0003] At present, the existing image structured processing methods usually divide the detection and attribute analysis of objects such as pedestrians or vehicles into two separate steps. For example, to detect and analyze the attributes of pedestrians in a picture, first detect and identify the objects in the image, mark the objects containing the target and the pedestrians in the image, and then extract features from all objects. However, this method may cause great errors in the feature extraction of the target recognition machine in an environment where a large number of pedestrian objects may cover each other or the light is poor, and lose important information, or recognize wrong information, causing deviations during retrieval. Therefore, traditional image structured recognition processing methods are greatly affected by environmental factors. Summary of the invention

[0004] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a method for extracting structured information from an image to solve the technical problem that the image structured recognition processing method is interfered by environmental factors, thereby reducing the error in target feature recognition.

[0005] In order to achieve the above purpose / solve the above technical problems, the present invention is implemented by adopting the following technical solutions:

[0006] In a first aspect, a method for extracting structured information from an image comprises:

[0007] Selecting an original image, wherein the original image has target object and background information;

[0008] Segmenting the target object and background information in the original image according to a pre-built semantic segmentation network model to extract the target object;

[0009] Based on the target object, according to a pre-built target detection model, the target object is identified to obtain target features in the target object;

[0010] The target features are analyzed by using a histogram to solve the connected domain method to obtain feature information in the target features.

[0011] In combination with the first aspect, further, the segmentation process of the original image includes:

[0012] Segmenting the original image into a plurality of pixels using the semantic segmentation network model;

[0013] Classify the plurality of pixel points to obtain a first pixel point and a second pixel point;

[0014] The first pixel point is a pixel point corresponding to the target object, and the second pixel point is a pixel point corresponding to the background information.

[0015] In combination with the first aspect, further, the encoder includes a backbone network and a spatial pyramid pooling layer with dilated convolution, for extracting scale features and depth features of the original image;

[0016] The decoder is used to receive the depth features and scale features from the encoder;

[0017] Among them, the deep features include low-level features and high-level features, and the scale features and depth features together constitute the basic image features of the original image.

[0018] In combination with the first aspect, further, the basic image feature can extract multiple scale features of different depths, and fuse the multiple scale features of different depths to obtain multi-scale features;

[0019] Among them, multi-scale features include low-level features of different scales and high-level features of different scales.

[0020] In combination with the first aspect, further, the decoding process of the decoder includes:

[0021] The decoder receives low-level features of the original image acquired by the backbone network and high-level features of different scales fused by a spatial pyramid pooling layer with dilated convolution, wherein the low-level features have image textures, and the multi-scale high-level features have semantic information.

[0022] In combination with the first aspect, further, the target detection model has a target recognition algorithm and a trained feature detection model, the target recognition algorithm is used to recognize the target object, and the feature detection model is used to detect the target features.

[0023] In combination with the first aspect, further, the process of obtaining feature information in the target feature includes:

[0024] Identify the target object according to the target recognition algorithm, and obtain attribute information of the target object;

[0025] Locking the target features based on the attribute information in the target object and the trained feature detection model;

[0026] The locked target features are analyzed by a method of solving connected domains with a histogram to obtain feature information in the target features;

[0027] Among them, the target feature is the local feature of the target object.

[0028] In a second aspect, a device for extracting structured information from an image includes:

[0029] An image selection module is used to select an original image, wherein the original image has target object and background information;

[0030] An image segmentation module, used to segment the target object and background information in the original image according to a pre-built semantic segmentation network model to extract the target object;

[0031] A feature recognition module is used to recognize the target object based on the target object and according to a pre-built target detection model, and obtain target features in the target object;

[0032] The feature analysis module is used to analyze the target feature by using a method of solving a connected domain using a histogram, and obtain feature information in the target feature.

[0033] In a third aspect, an electronic terminal comprises a processor and a memory connected to the processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the steps of the method as described in any one of the first aspects are performed.

[0034] In a fourth aspect, a computer-readable storage medium stores a computer program thereon, wherein the program, when executed by a processor, implements the steps of any one of the methods described in the first aspect.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] The original image is segmented through a pre-built semantic segmentation network model to obtain the target object and background information of the original image after segmentation, thereby achieving the purpose of target object extraction; based on the extracted target object, the target features in the target object are identified through a pre-built target detection model, and finally the identified target features are analyzed through the histogram solution of the connected domain method to obtain the specific feature information in the target features. After layers of identification and analysis, the interference of environmental factors is reduced to a certain extent, thereby reducing the error of target feature recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 is a flow chart of a method for extracting structured image information provided by an embodiment of the present invention;

[0038] Figure 2 It is a schematic diagram of the image segmentation process of the Deeplabv3p network provided by an embodiment of the present invention;

[0039] Figure 3 It is a schematic diagram of the YoloV3 part detection principle provided by an embodiment of the present invention;

[0040] Figure 4 It is a schematic diagram of the structure of an image structured information extraction device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The technical solution of the present invention is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present application and the specific features in the embodiments are detailed descriptions of the technical solution of the present application, rather than limitations on the technical solution of the present application. In the absence of conflict, the embodiments of the present application and the technical features in the embodiments can be combined with each other.

[0042] The term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. Embodiment 1

[0043] Figure 1 This is a flow chart of a method for extracting structured information from an image in Embodiment 1 of the present invention. This flow chart only shows the logical sequence of the method described in this embodiment. In other possible embodiments of the present invention, different methods may be used without conflict. Figure 1 The steps shown or described are accomplished in the order shown.

[0044] The image structured information extraction method provided in this embodiment can be applied to a terminal and can be executed by an image structured information extraction device, which can be implemented by software and / or hardware, and can be integrated in a terminal, such as any smart phone, tablet computer or computer device with communication function. Figure 1 The method of this implementation specifically includes the following steps:

[0045] Step S110: Select an original image with target object and background information, and the original image can be obtained by an image acquisition device such as a camera or a webcam;

[0046] Step S120: segmenting the target object and background information in the original image according to the pre-built semantic segmentation network model to extract the target object;

[0047] It should be noted that the semantic segmentation network model adopted in the embodiment of the present invention is DeeplabV3+. The purpose of semantic segmentation is to classify several pixels in the selected original image and divide them into first pixels and second pixels for the final pixel-by-pixel classification, that is, a pixel-by-pixel classification method is used to equip each pixel in the image with a class label. In this case, each pixel is processed independently to improve the accuracy of subsequent image recognition.

[0048] Among them, the first pixel point is the pixel point corresponding to the target object; the second pixel point is the pixel point corresponding to the background information.

[0049] Specifically, the image segmentation algorithm based on the Deeplabv3p network segments the selected original image to separate the target object and background information, which is convenient for extracting the required target object from the original image. The target object finally segmented can be set as the image foreground, so that the target object is located in front of the image background, and the image background is set to zero to prevent interference from environmental factors in subsequent target feature detection.

[0050] It should be noted that if the original image after segmentation is found to not have the target object, it is necessary to reselect the image until the selected image can have the target object.

[0051] See also Figure 2 , DeeplabV3+ has at least an encoder (Encoder) and a decoder (Decoder). The encoder is used to extract the scale features and depth features of the original image, and the encoder is composed of a backbone network (DCNN) and a spatial pyramid pooling layer (ASPP) with hole convolution; the decoder is used to receive the scale features and depth features from the encoding.

[0052] It should be noted that the depth features include low-level features and high-level features, and the scale features and depth features together constitute the basic image features of the original image.

[0053] Specifically, the backbone network (DCNN) adopts common structures such as residual networks based on basic image features. Based on this common structure, the spatial pyramid pooling layer with dilated convolution extracts multiple scale features of different depths and fuses multiple scale features of different depths to achieve multi-scale feature extraction.

[0054] Among them, multi-scale features include low-level features of different scales and high-level features of different scales.

[0055] Furthermore, the decoder receives low-level features from the backbone network (DCNN) and high-level features of different scales fused through a spatial pyramid pooling layer with dilated convolutions. The low-level features carry basic information such as image texture, and the high-level features of different scales carry semantic information. The high- and low-level features are fused after upsampling to improve the accuracy of the target object.

[0056] Step S130: Based on the target object, the target object is identified according to a pre-built target detection model to obtain target features in the target object;

[0057] It should be noted that the target detection model used in the embodiment of the present invention is the YoloV3 network, which is a single-stage end-to-end target detection model. The YoloV3 network has higher accuracy and speed.

[0058] Taking the identification of road surrounding maps as an example, the target objects are identified through the target recognition algorithm in the YoloV3 network. The identified target objects include pedestrians and vehicles. After the target objects are determined, the target features of the target objects are detected through the feature detection model trained in the YoloV3 network, such as whether pedestrians are wearing hats, whether pedestrians are wearing masks, whether the vehicle type is an SUV or a sedan, etc.

[0059] See also Figure 3 The backbone network (BackBone) of the YoloV3 network uses the DarkNet53 deep model as the basic feature extraction model. Features of different levels are extracted at different stages of the backbone network (BackBone), and upsampled and fused from deep to shallow (bottleneck layer) to obtain outputs of three different scales. The specific output scale depends on the actual situation.

[0060] Specifically, in an embodiment of the present invention, a feature detection model is added on the basis of a YoloV3 network. The feature detection model takes the output of the bottleneck layer of the YoloV3 network as input, converts the pedestrian detection result area in the output result into a region of interest (ROL), and uses the alignment layer in the region of interest to extract the target features in the bottleneck layer. Taking a road perimeter map as an example, the pedestrian area features in the map are further processed. Considering that the images of pedestrians are often in a vertical direction, a horizontal average pooling operation is performed on the target features to obtain features of three different scales such as 1×19, 1×38, and 1×76. The features of these three different scales are processed by a fully connected layer and the prediction results are output.

[0061] It should be noted that the mask area and clothing area in the prediction result are considered to occupy the horizontal direction and are only distinguished in the vertical direction.

[0062] Step S140: Analyze the target feature by using a histogram to solve the connected domain, and obtain feature information in the target feature.

[0063] The target object is identified according to the target recognition algorithm to obtain the attribute information of the target object; the target features are locked based on the attribute information of the target object and the trained feature detection model; the locked target features are analyzed by a histogram solution connected domain method to obtain the feature information in the target features.

[0064] Among them, the target feature is the local feature of the target object.

[0065] Specifically, taking the road perimeter map as an example, it is detected that the target object's clothes color is red, that is, the pedestrian is the target object, the clothes are the target feature, and the clothes color is the feature information. Based on the segmented pedestrian foreground area and the pedestrian detection and part detection results, a foreground area of ​​a pedestrian part (mask, hat, top, pants, etc.) is obtained, the color histogram of the area is counted, and the color median is calculated to determine the color of the target part. Embodiment 2

[0066] See also Figure 4 , an embodiment of the present invention provides an image structured information extraction device, comprising:

[0067] An image selection module is used to select an original image, wherein the original image has target object and background information;

[0068] An image segmentation module, used to segment the target object and background information in the original image according to a pre-built semantic segmentation network model to extract the target object;

[0069] A feature recognition module is used to recognize the target object based on the target object and according to a pre-built target detection model, and obtain target features in the target object;

[0070] The feature analysis module is used to analyze the target feature by using a method of solving a connected domain using a histogram, and obtain feature information in the target feature.

[0071] For other technical features not covered in this embodiment, please refer to Embodiment 1. Embodiment 3

[0072] An embodiment of the present invention further provides an electronic terminal, including a processor and a storage medium;

[0073] The storage medium is used to store instructions;

[0074] The processor is used to operate according to the instructions to execute the steps of the method described in embodiment 1. Embodiment 4

[0075] The present invention also provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of the method described in embodiment 1 are implemented.

[0076] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing related hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory.

[0077] By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0078] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0079] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0080] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0081] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for extracting structured information from an image, characterized in that: include: Selecting an original image, wherein the original image has target object and background information; Segmenting the target object and background information in the original image according to a pre-built semantic segmentation network model to extract the target object; The segmented target object is set as the image foreground, so that the target object is in front of the image background, and the image background is set to zero; Based on the target object, the target object is identified according to a pre-built target detection model to obtain target features in the target object, wherein the target detection model has a target recognition algorithm and a trained feature detection model, wherein the target recognition algorithm is used to identify the target object, and the feature detection model is used to detect the target features; The feature detection model uses the output of the bottleneck layer of the target detection network as input, converts the target detection result area in the output result into a region of interest, and uses the alignment layer in the region of interest to extract the target features in the bottleneck layer; The target feature is analyzed by using a method of solving a connected region using a histogram to obtain feature information in the target feature. The process of obtaining the feature information in the target feature includes: The target object is identified according to the target recognition algorithm to obtain the attribute information of the target object; the target feature is locked based on the attribute information of the target object and the trained feature detection model; the locked target feature is analyzed by a histogram solution connected domain method to obtain the feature information of the target feature; wherein the target feature is a local feature of the target object.

2. The method for extracting structured image information according to claim 1, characterized in that: The segmentation process of the original image includes: Segmenting the original image into a plurality of pixels using the semantic segmentation network model; Classify the plurality of pixel points to obtain a first pixel point and a second pixel point; The first pixel point is a pixel point corresponding to the target object, and the second pixel point is a pixel point corresponding to the background information.

3. The method for extracting structured image information according to claim 2, characterized in that: The semantic segmentation network model has at least an encoder and a decoder; The encoder includes a backbone network and a spatial pyramid pooling layer with dilated convolutions, which is used to extract scale features and depth features of the original image; The decoder is used to receive the depth features and scale features from the encoder; Among them, the deep features include low-level features and high-level features, and the scale features and depth features together constitute the basic image features of the original image.

4. The method for extracting structured image information according to claim 3, characterized in that: The basic image features can extract multiple scale features of different depths, and fuse the multiple scale features of different depths to obtain multi-scale features; Among them, multi-scale features include low-level features of different scales and high-level features of different scales.

5. The method for extracting structured image information according to claim 4, characterized in that: The decoding process of the decoder includes: The decoder receives low-level features of the original image acquired by the backbone network and high-level features of different scales fused by a spatial pyramid pooling layer with dilated convolution, wherein the low-level features have image textures, and the multi-scale high-level features have semantic information.

6. An image structured information extraction device, characterized in that: include: An image selection module is used to select an original image, wherein the original image has target object and background information; An image segmentation module, used to segment the target object and background information in the original image according to a pre-built semantic segmentation network model to extract the target object; The segmented target object is set as the image foreground, so that the target object is in front of the image background, and the image background is set to zero; A feature recognition module is used to recognize the target object based on the target object and according to a pre-built target detection model, and obtain the target features in the target object. The target detection model has a target recognition algorithm and a trained feature detection model. The target recognition algorithm is used to recognize the target object, and the feature detection model is used to detect the target features. Among them, the feature detection model uses the output of the bottleneck layer of the target detection network as input, converts the target detection result area in the output result into a region of interest, uses the alignment layer in the region of interest to extract the target features in the bottleneck layer, and uses the alignment layer in the region of interest to extract the target features in the bottleneck layer; The feature analysis module is used to analyze the target feature by using a method of solving the connected domain using a histogram, and obtain feature information in the target feature. The process of obtaining the feature information in the target feature includes: The target object is identified according to the target recognition algorithm to obtain the attribute information of the target object; the target feature is locked based on the attribute information of the target object and the trained feature detection model; the locked target feature is analyzed by a histogram solution connected domain method to obtain the feature information of the target feature; wherein the target feature is a local feature of the target object.

7. An electronic terminal, characterized in that: The method comprises a processor and a memory connected to the processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the steps of the method according to any one of claims 1 to 5 are performed.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 5 are implemented.

Images