Image annotation method, target detection method and device, and storage medium

Abstract

The present application relates to an image annotation method, a target detection method and device, and a storage medium, wherein the image annotation method includes: obtaining an image to be annotated, initially annotating the image to be annotated, and obtaining first annotation information of the detection target; analyzing second annotation information of the detection target within a preset search range, and generating a training sample image using the first annotation information and the second annotation information. The present application makes full use of graph search technology in the automatic image annotation process, and easily obtains the affine transformation relationship between the standard template and the candidate template, thereby automatically annotating the missing nodes on the detection target, which can improve the efficiency of automatic image annotation and enhance the stability of the subsequent target detection algorithm.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image annotation method, a target detection method and device, and a storage medium. Background Art

[0002] In recent years, artificial intelligence and big data have become the focus of attention in various fields at home and abroad. Faced with more and more image data, how to efficiently manage and organize these image data has become a hot issue in the field of image retrieval and target recognition. Image annotation, which adds text information related to the image content, has become the most important solution. In view of the problems of large workload and strong subjectivity of manual annotation, intelligent image annotation has attracted the favor of many researchers under the wave of artificial intelligence.

[0003] Most of the automatic image annotation techniques select appropriate labels from the keyword dictionary to annotate the image to be annotated by exploring the semantic content of the image or exploring the similarity between the visual features of the image and the semantic features. The key technology lies in establishing the connection between low-level features and high-level semantic labels and position information. The existing automatic image annotation methods based on generative models annotate keywords for the images to be annotated by calculating the joint probability of image features and labels in the dictionary. There are some problems with this annotation method, such as the inability to guarantee the optimality of the predicted labels and the difficulty of using generative models to capture the complex relationship between image features, position information, affine transformation relationships and labels. Most automatic image annotation algorithms annotate the images to be annotated with a fixed number of labels, but in reality, the number of labels for different images is not fixed. Annotating each image with the same number of labels will result in problems such as some images with rich content not being fully represented, while some images with relatively simple content are annotated with too many labels. Therefore, this unified annotation method will affect the accuracy of the annotation model.

[0004] Automatic image annotation mainly solves two problems: one is the category problem and the other is the location problem. At present, the location problem is basically solved by obtaining only the coordinate information of the location, among which the angle, scaling and other information are not obtained, and the subsequent processing lacks the prior knowledge of the location information. At present, most of the location and category information are obtained by establishing the mapping relationship of key feature points through traditional features, such as extracting the SIFT features of image data and the SVM classifier trained by the HOG features, and combining the neural network classifier built by color and texture features to automatically annotate the image data. However, the classifier trained by shallow image features is only effective for annotating some targets, and cannot obtain high annotation accuracy for other targets. Another method is to provide annotation information through feature correlation analysis, such as extracting high-level features of image data through a deep Boltzmann machine model, and screening out the best annotation of image data through methods such as canonical correlation analysis, but the training of the Boltzmann model requires the use of a large number of training samples to obtain shallow visual features. Due to the different feature descriptions of different visual targets, it is often necessary to select better visual feature descriptions in a targeted manner, resulting in poor portability of this method. Summary of the invention

[0005] The main technical problem solved by the present invention is how to improve the accuracy and stability of target detection in an image. To solve the above technical problem, the present application discloses an image annotation method, a target detection method and device, and a storage medium.

[0006] According to the first aspect, an embodiment provides an image annotation method, which includes: acquiring an image to be annotated; the image to be annotated includes at least one detection target; performing initial annotation on the image to be annotated to obtain first annotation information of the detection target; the first annotation information includes relative position information and search range information of several nodes on the detection target; analyzing second annotation information of the detection target within a preset search range; the second annotation information includes semantic information, position information and scale information of missing nodes on the detection target; and generating a training sample image using the first annotation information and the second annotation information.

[0007] The initial annotation of the image to be annotated to obtain the first annotation information of the detection target includes: selecting a number of nodes on the detection target; the nodes are used to characterize the surface features of the detection target; and annotating and generating relative position information and search range information of the several nodes, thereby forming the first annotation information.

[0008] The method of analyzing the second annotation information of the detection target within a preset search range includes: determining the search range of each of the plurality of nodes according to a preset standard template; the search range of each node includes the search angle, search area, clipping range and search scale of the node; searching for other nodes on the detection target within the search range of each node, and using the other nodes to form a candidate template; and performing a speculative analysis on the missing nodes of the candidate template according to the standard template to obtain the second annotation information of the detection target.

[0009] The construction process of the standard template includes: obtaining a standard part image of the standard part corresponding to the detection object, setting a plurality of nodes on the standard part image for characterizing the surface features of the standard part; configuring a search range of the standard part corresponding to the detection target on the standard part image; the search range on the standard part image includes a search angle, a shearing angle, a search area and a search scale of the standard part; calculating the relative position between every two nodes on the standard part to obtain edge information of every two nodes; generating a graph structure of the standard part using every two nodes and corresponding edge information; and constructing the standard template according to the search range of the standard part image and the graph structure of the standard part.

[0010] The step of performing a speculative analysis on the missing nodes of the initial template according to the standard template to obtain second annotation information of the missing nodes on the detection target includes: using G = (V1, A) to represent the standard template, and using H t ={(V t ,B)} represents the set of candidate templates; where V1 is the set of nodes on the standard part, A is the set of edge information between nodes on the standard part, V t is the set of nodes on the detection target corresponding to the t-th candidate template, and B is the set of edge information between the nodes on the detection target corresponding to the t-th candidate template; the relative positions between the nodes are determined according to the set A in G=(V1,A), and H=(V1,A) is inferred through the template pose affine transformation. t , B) nodes with similar positions, thereby obtaining the missing nodes on the detection target; generating semantic information, position information and scale information of the missing nodes, constructing the connection relationship between the edges of the nodes according to the labeled nodes and the missing nodes, and obtaining the candidate template corresponding to the highest confidence, thereby forming the second annotation information of the missing nodes.

[0011] The relative positions of the nodes are determined according to the set A in G = (V1, A), and H is inferred by the template pose affine transformation. t ={(V t, B)}, thereby obtaining the missing nodes on the detection target, including: determining the relative positions of the nodes in the set A in G=(V1,A) and H t ={(V t ,B)}, the relative positions of the nodes determined by the set B are compared to infer the similarity relationship of the node positions, which can be expressed as

[0012]

[0013] Where n is the number of nodes in set B, i is the index number of the node, k is the number of instances of the i-th node in the image to be labeled, δ is the error and is expressed as

[0014]

[0015] Wherein, M represents the standard image, E represents the image to be annotated, j is the index number, and ψ is the back-projection error function; when the sum of the reflected projection error reaches the minimum, it is determined The corresponding candidate template, where b ij is the similarity of the nodes in set B; find the missing nodes on the detection target according to the determined score value of the candidate template; the score value is expressed as

[0016]

[0017] Among them, d i is a process parameter and satisfies d i =||p i -(Rq i +t)|| 2 , p i is the coordinate vector of the i-th node in the standard template, q i is the coordinate vector of the i-th node in the determined candidate template, R is the rotation transformation relationship of the coordinate vector, t is the translation transformation relationship of the coordinate vector, n and m represent the total number of nodes on the detection target and the number of labeled nodes respectively.

[0018] According to the second aspect, an embodiment provides a target detection method based on graph matching, which includes: acquiring an image to be detected; the image to be detected includes at least one detection target; detecting the image to be detected according to a preset image detection model to obtain node information of each node on the detection target and template information of the detection target; the image detection model is obtained by training a network model using the training sample images obtained by the method in the first aspect above; outputting the node information and the template information; the node information includes semantic information, position information and scale information, and the template information includes the posture change of the detection target in terms of angle, area and scale.

[0019] According to the third aspect, an embodiment provides a target detection device, which includes: an image acquisition component, used to acquire an image of at least one detection target in a scene and obtain an image to be detected; a processor, connected to the image acquisition component, used to obtain node information of each node on the detection target according to the target detection method described in the second aspect above, and / or obtain an identification type of the detection target; a display, connected to the processor, used to display the image to be detected, the node information of each node on the detection target and / or the identification type of the detection target.

[0020] The processor includes a training module and a detection module; the training module is used to train a network model using one or more training sample images, and obtain an image detection model by updating network parameters; the training sample images are training sample images obtained by the method in the first aspect above; the detection module is connected to the training module, and is used to input the image to be detected into the image detection model and perform detection processing, and output node information of each node on the detection target, template information of the detection target, and / or recognition type of the detection target.

[0021] According to the fourth aspect, an embodiment provides a computer-readable storage medium, which includes a program, and the program can be executed by a processor to implement the image annotation method described in the first aspect above, or to implement the target detection method described in the second aspect above.

[0022] The beneficial effects of this application are:

[0023] According to the above-mentioned embodiment, an image annotation method, a target detection method and device, and a storage medium, wherein the image annotation method includes: obtaining an image to be annotated, initially annotating the image to be annotated, and obtaining first annotation information of the detection target; analyzing second annotation information of the detection target within a preset search range, and generating a training sample image using the first annotation information and the second annotation information. The target detection method includes: obtaining an image to be detected, detecting the image to be detected according to a preset image detection model, obtaining node information of each node on the detection target and template information of the detection target, and the image detection model is obtained by training a network model using the training sample image obtained in the image annotation method; and outputting the node information and the template information. First, in the image annotation method, since the image to be annotated is initially annotated to obtain the first annotation information of the detection target, more prior knowledge is provided for subsequent image annotation, so that the subsequent annotation can be quickly and automatically annotated according to the relative position relationship of the annotated nodes, thereby improving the annotation efficiency; second, since the second annotation information of the detection target is analyzed within the preset search range, the automatic annotation process of the detection target makes full use of the graph search technology, and it is easy to obtain the affine transformation relationship between the standard template and the candidate template, so as to automatically annotate the missing nodes on the detection target, which can improve the efficiency of automatic image annotation; third, in the target detection method, since a network model is trained using the annotated training sample images, Obtaining an image detection model makes the image detection model have high stability and accuracy, and can accurately output the node information of each node on the detection target and the template information of the detection target; fourthly, in the present application, the posture problem is solved based on the preset standard template and graph search technology, and the category problem is solved by using deep learning target detection technology, thereby decomposing the graph matching problem into two problems to be solved, and then converted into a graph matching algorithm for solution, which helps to improve the stability of graph matching; fifthly, in the process of node search and posture matching, the present application adopts a hierarchical matching strategy and template pose affine transformation, so it is easy to obtain the template with the highest score as the final matching posture, thereby reducing the complexity of the template connection relationship and improving the algorithm operation efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is a flowchart of the image annotation method in Embodiment 1 of the present application;

[0025] Figure 2 Flowchart for building standard templates;

[0026] Figure 3 A flow chart for obtaining first labeling information of a detection target;

[0027] Figure 4 A flow chart for obtaining second labeling information of a detection target;

[0028] Figure 5 This is a schematic diagram of the principle of marking nodes on the bolt as the inspection target;

[0029] Figure 6 The schematic diagram is a principle diagram for finding missing nodes on the bolt inspection target;

[0030] Figure 7 This is a flow chart of a target detection method based on graph matching in Embodiment 2 of the present application;

[0031] Figure 8 This is a schematic diagram of the structure of the target detection device in the third embodiment of the present application;

[0032] Fig. 9 is a schematic diagram of the structure of the processor;

[0033] Fig.10 It is a working principle diagram of the processor in each processing stage;

[0034] Fig.11 This is a schematic diagram of the structure of the target detection device in Example 4 of the present application. DETAILED DESCRIPTION

[0035] The present invention is further described in detail below by specific embodiments in conjunction with the accompanying drawings. Wherein similar elements in different embodiments adopt associated similar element numbers. In the following embodiments, many detailed descriptions are for making the present application better understood. However, those skilled in the art can easily recognize that some features can be omitted in different situations, or can be replaced by other elements, materials, methods. In some cases, some operations related to the present application are not shown or described in the specification, this is to avoid the core part of the present application being overwhelmed by too much description, and for those skilled in the art, it is not necessary to describe these related operations in detail, and they can fully understand the related operations according to the description in the specification and the general technical knowledge in the art.

[0036] In addition, the features, operations or characteristics described in the specification can be combined in any appropriate manner to form various implementations. At the same time, the steps or actions in the method description can also be interchanged or adjusted in a manner that is obvious to those skilled in the art. Therefore, the various sequences in the specification and the drawings are only for the purpose of clearly describing a certain embodiment and are not meant to be a required sequence, unless otherwise specified that a certain sequence must be followed.

[0037] The serial numbers of the components in this document, such as "first", "second", etc., are only used to distinguish the objects described and do not have any order or technical meaning. The "connection" and "coupling" mentioned in this application, unless otherwise specified, include direct and indirect connections (couplings).

[0038] The inventive concept of the technical solution of this application is: in order to speed up the image annotation speed, a small amount of initial annotation is performed in advance in the image to be annotated, and the format of the annotation information is made the same as the format of the node information, so that the search range of each node can be calculated according to the search range of the standard template, and other missing nodes are found within the search range of each node to form a candidate template; after finding the candidate template, the annotation information of the remaining missing nodes is calculated according to the relative position information between the nodes of the standard template, so as to achieve the automatic annotation function of large-scale data sets through a small amount of initial annotation. This application adopts a combination of manual annotation and automatic annotation to realize large-scale data set annotation, provides technical support for subsequent target detection processing, and is conducive to improving the performance of target detection.

[0039] Embodiment 1

[0040] The present application discloses an image annotation method, which is mainly applied on an image processing device to realize automatic annotation processing of an image to be annotated.

[0041] In this embodiment, please refer to Figure 1 ,The image annotation method includes steps S110-S140, which are respectively described below.

[0042] Step S110, obtaining an image to be annotated. The image to be annotated here is used to participate in network training after the image is annotated, so the image to be annotated should be a picture of a certain detection scene, and the image should include at least one detection target; for example, when it is necessary to detect and identify various types of bolts in the scene, the image to be annotated should include at least one bolt.

[0043] Of course, network training cannot be completed using a single image to be labeled, so in this embodiment, multiple images to be labeled can be obtained and image annotation can be performed separately to generate multiple training sample images. Since the annotation method for each image to be labeled is the same, this embodiment only describes in detail the annotation method for a single image to be labeled.

[0044] Of course, if the image to be annotated includes multiple detection targets, then these detection targets need to be annotated separately, and a lot of annotation information will be generated. Since each detection target can be annotated in the same way, this embodiment will only describe in detail the annotation process of a single detection target.

[0045] It should be noted that the detection target in the image to be labeled is the imaging state of a certain type of object. If various types of bolts are used as detection targets, only the bolts in the image to be labeled need to be labeled.

[0046] Step S120, initially annotate the image to be annotated to obtain first annotation information of the detection target. The initial annotation here can be performed by manual annotation, machine annotation, etc., and the first annotation information obtained by annotation includes relative position information and search range information of several nodes on the detection target.

[0047] It should be noted that in order to reduce the workload of image annotation, key pixel points and key pixel areas on the image are often annotated. Therefore, nodes can be used to represent the surface features of the detection target, and the annotated nodes are usually the display results of those expressive features on the detection target.

[0048] for example Figure 5 , the image to be detected includes a bolt 1, then a pixel point or a pixel area can be selected at the head, middle and tail of the bolt respectively as several nodes on the bolt, so that the processor calculates and generates the phase position information and search range information of nodes C, A and B.

[0049] Step S130: Analyze and obtain the second annotation information of the detection target within the preset search range. The analysis process can be performed by automatic machine annotation, and the second annotation information obtained by annotation includes semantic information, position information and scale information of the missing nodes on the detection target.

[0050] It should be noted that although several nodes on the detection target in the image to be detected can be marked by manual marking, there are still many nodes on the detection target that have not been found. In this case, the nodes that have not been found are regarded as missing nodes on the detection target.

[0051] For any missing node, its semantic information is the confidence of the image feature it represents in each category, usually represented by one-hot encoding, its position information is the coordinates of the center point of the image feature it represents on the image (including x-axis coordinates and y-axis coordinates), and its scale information is the size occupied by the image feature it represents on the image (such as the number of pixels in length and width).

[0052] Step S140, generating a training sample image using the first annotation information and the second annotation information. Since the missing nodes on the detection target also represent some key image features (or surface features) on the detection target, and the number is large, and like the annotated nodes on the detection target, they can reflect most of the posture information and texture information of the detection target on the image to be annotated, then after adding the first annotation information and the second annotation information to the image to be annotated, the image annotation work of the image to be annotated can be completed, thereby generating a training sample image.

[0053] In this embodiment, the process of automatic image annotation requires the use of a standard template to perform inference analysis of missing nodes, so the process of constructing the standard template will be described in advance. Figure 4 ,The construction process of the standard template can be represented by steps S210-S240, which are specifically described as follows.

[0054] Step S210, obtaining a standard part image of a standard part corresponding to the object to be detected, and setting a plurality of nodes for characterizing the surface features of the standard part on the standard part image. Setting a plurality of nodes for characterizing the surface features of the standard part can be set by manual annotation or by automatic annotation, which is not specifically limited here.

[0055] Step S220, configuring a search range of the standard part corresponding to the detection target on the standard part image. Here, the search range on the standard part image includes a search angle, a shear angle, a search area and a search scale of the standard part.

[0056] It should be noted that the standard parts corresponding to the test targets refer to the standard reference objects of the test targets. For example, for the test target of bolts, they may present different imaging states in different scenes, and even have deformed, unclear, or obstructed imaging conditions. However, the standard parts image of the standard parts corresponding to the bolts is in a complete imaging state and can be used as the basic reference form of the bolts.

[0057] For the search range configured on the standard part image, the search angle refers to the rotation angle variation range of each image feature on the standard part on the image, the shear angle refers to the shear deformation angle of each image feature on the standard part in the x direction or y direction, the search area refers to the area range where each image feature on the standard part may appear on the image, and the search scale refers to the scaling range of each image feature on the standard part on the image.

[0058] Step S230, calculating the relative position between every two nodes on the standard part to obtain the edge information of every two nodes; and generating the graph structure of the standard part using every two nodes and the corresponding edge information.

[0059] It should be noted that the edge information between every two nodes can be the relative position information of the two nodes, such as the angle between the straight line where the two nodes are located and the x-axis, the difference in x-coordinates, the difference in y-coordinates, the Euclidean distance and other values.

[0060] Step S240: constructing a standard template according to the search range of the standard part image and the graph structure of the standard part.

[0061] In this embodiment, please refer to Figure 3 The above step S120 mainly involves the process of labeling and obtaining the first labeling information on the detection target, which specifically includes steps S121-S122, which are described as follows.

[0062] Step S121, selecting a plurality of nodes on the detection target, wherein each selected node is used to characterize the surface feature of the detection target.

[0063] for example Figure 5 , the user manually clicks on the head, middle and tail of the bolt 1 in the image to be annotated, so that the processor recognizes that the pixel areas at these three positions are selected, thereby obtaining the coordinates of the center of gravity of the selected area, and then easily determining that nodes C, A, and B are selected on the detection target, and as several selected nodes. Among them, node C represents the image feature of the bolt head, node A represents the image feature of the middle part of the bolt, and node B represents the image feature of the bolt tail.

[0064] Step S122: annotate and generate relative position information and search range information of a plurality of nodes, thereby forming first annotation information.

[0065] for example Figure 5 Since the processor has known the coordinates of several nodes (i.e., nodes C, A, and B), it is easy to calculate the angle between the straight line where each two nodes are located and the x-axis, the difference in x-coordinates, the difference in y-coordinates, the Euclidean distance, and other values ​​based on the coordinates as the relative position information of the two nodes. In addition, the search range information of each node can be pre-set by the user and input parameters through the interactive interface.

[0066] It should be noted that the search range information may specifically include a search angle, a shearing angle, a search area, and a search scale. The search angle refers to the range of rotation angles of the image feature represented by the node on the image, the shearing angle refers to the shear deformation angle of the image feature represented by the node in the x-direction or y-direction, the search area refers to the area where the image feature represented by the node may appear on the image, and the search scale refers to the zoom range of the image feature represented by the node on the image. Since the search range information is a common setting parameter for image annotation, it will not be described in detail here.

[0067] Those skilled in the art will appreciate that, since the image to be annotated is initially annotated to obtain the first annotation information of the detection target, more prior knowledge can be provided for subsequent image annotation, so that subsequent annotation can be quickly and automatically performed based on the relative position relationship of the annotated nodes, thereby improving the annotation efficiency.

[0068] In this embodiment, please refer to Figure 4 The above step S130 mainly involves the process of labeling and obtaining the second labeling information on the detection target, which specifically includes steps S131-S133, which are described as follows.

[0069] Step S131, determining the search range of each node among a plurality of nodes according to a preset standard template.

[0070] The preset standard template here can be the sum of the image features of all nodes on the standard part corresponding to the detection target, including the search range of all nodes and the graph structure composed of every two nodes. Since the standard template includes the search range of all nodes, the search range of each node in the detection target in the image to be annotated can be set according to the standard template, so that the setting parameters of the search range remain consistent. For the construction process of the standard template, please refer to steps S210-S240 above.

[0071] Of course, the search range of each node includes the search angle, search area, clipping range and search scale of the node. Figure 5 ,For nodes C, A, and B on the bolt, the outer rectangular dotted border of each node can be used as the search area of ​​the node.

[0072] Step S132, searching for other nodes on the detection target within the search range of each node, and using other nodes to form candidate templates.

[0073] for example Figure 5 In order to find other nodes on the bolt, we can start from nodes C, A, and B respectively, and expand the search for the remaining pixels in the corresponding search area. When other expressive image features are found, the pixel points or pixel areas corresponding to the image features are used as other nodes, so that the other nodes found can be used to form candidate templates.

[0074] It should be noted that, if there are multiple detection targets in the image to be detected, since these detection targets are all annotated, multiple candidate templates will be formed at this time, thereby forming a set of candidate templates.

[0075] Step S133, inferring and analyzing the missing nodes of the candidate template according to the standard template to obtain the second annotation information of the detection target. Since the standard template includes information of all nodes on the detection target and the candidate template only includes information of a part of the nodes, the standard template can be used to infer and analyze the missing nodes in the candidate template.

[0076] It should be noted that, since the second annotation information of the detection target is analyzed within the preset search range, the automatic annotation process of the detection target makes full use of the graph search technology, which makes it easy to obtain the affine transformation relationship between the standard template and the candidate template, thereby automatically annotating the missing nodes on the detection target, which can improve the efficiency of automatic image annotation.

[0077] In a specific embodiment, step S133 specifically includes the following three sub-steps:

[0078] (1) In order to facilitate the numerical description of the inference analysis process, the standard template can be represented by G = (V1, A) and H t ={(V t ,B)} represents the set of candidate templates; where V1 is the set of nodes on the standard parts corresponding to the detection target, A is the set of edge information between nodes on the standard parts, and V t is the set of nodes on the detection target corresponding to the t-th candidate template, and B is the set of edge information between nodes on the detection target corresponding to the t-th candidate template.

[0079] (2) Determine the relative positions of the nodes based on the set A in G = (V1, A), and infer H = (V1, A) through the affine transformation of the template pose. t ,B) nodes with similar position relationships, thereby obtaining the missing nodes on the detection target.

[0080] (3) Generate semantic information, position information, and scale information of the missing nodes, construct the edge connection relationship between the nodes based on the labeled nodes and the missing nodes, obtain the candidate template corresponding to the highest confidence, and thus form the second annotation information of the missing nodes.

[0081] In the above sub-step (2), some complex calculations are involved, which will be explained in detail here. In the template pose affine transformation, the set A in G = (V1, A) determines the relative positions between the nodes and H t ={(V t ,B)}, the relative positions of the nodes determined by the set B are compared to infer the similarity relationship of the node positions, which can be expressed as

[0082]

[0083] Where n is the number of nodes in set B, i is the index number of the node, k is the number of instances of the i-th node in the image to be labeled (if there are s detection targets in the image to be labeled, then the number of instances of node i is s); →min means taking the minimum value operation. In addition, δ is the error and is expressed as

[0084]

[0085] Wherein, M represents the standard part image of the standard part, E represents the image to be annotated, j is the index number, ψ is the back projection error function; arg min() represents the calculation of the variable value when the function takes the minimum value.

[0086] When the sum of the reflection and projection errors reaches the minimum (i.e., the formula is satisfied ) confirmed The corresponding candidate template, where b ij is the similarity of the nodes in set B.

[0087] Find the missing nodes on the detection target according to the determined score value of the candidate template; the score value is expressed as

[0088]

[0089] Among them, d i is a process parameter and satisfies d i =||p i -(Rq i +t)|| 2 , p i is the coordinate vector of the i-th node in the standard template and satisfies {p i ,i=1…n},q i is the coordinate vector of the i-th node in the determined candidate template and satisfies {q i , i=1…n}, R is the rotation transformation relationship of the coordinate vector, t is the translation transformation relationship of the coordinate vector; n and m represent the total number of nodes on the detection target and the number of marked nodes respectively.

[0090] It should be noted that if the influence of angle and length is taken into account, the back-projection error function ψ can be a monotonically increasing function. In this case, the back-projection error function can be expressed as

[0091]

[0092] Among them, δ pos , δ orient are position error and angle error respectively.

[0093] It should be noted that the information constituting the template includes the relative position relationship between nodes, and the weight between node i and node j represents the similarity between the node position relationship and the template. ij , so the template can be represented by a fully connected directed graph (V, B), where |V| = n, |B| = n(n-1), for each edge b in the set edge B ij ∈B, the corresponding weight is Ω ij .

[0094] To calculate the pose relationship, the pose transformation matrix is ​​updated each time a data point is added. To facilitate the solution of the pose transformation relationship, the objective function can be established:

[0095]

[0096] According to the objective function, the weight corresponding to each edge can be calculated, that is, the weighted representation of the connection relationship between nodes can be obtained. When expressing the weight, it is considered that the greater the distance from the template point, the lower the score should be, and the smaller the distance, the higher the score, and the score is set in the range of [0,1]. Considering that there are a total of n nodes and m nodes have been labeled, the score value (i.e. confidence) of the candidate template can be The nodes in the standard template are compared with a preset threshold, so as to determine which nodes in the candidate template are missing through the comparison result.

[0097] for example Figure 5 and Figure 6 For the bolt detection target, there may be multiple different bolts distributed in different postures in the image to be annotated. The head image feature of the bolt is represented by node C, the middle image feature is represented by node A, and the tail image feature is represented by nodes B, R, and P, and nodes B, R, and P represent different image features of the bolt tail respectively. Under normal circumstances, it is necessary to determine an optimal node connection method based on the relative position relationship of each node. If CAP, CAR, and CAB are to be accurately matched, a candidate template CA-BPR can be established, so that the third node fuzzy control can be achieved, thereby selecting one or more optimal models from these combinations. In addition, in the subsequent image matching process, each image feature can be determined in a similar manner based on the relative position relationship of each node, and the position transformation constraint is satisfied, so that a template is constructed by the node features. In this case, the number of nodes to be searched can be reduced, and the interference of redundant nodes can be avoided, thereby improving the stability and accuracy of detection.

[0098] Embodiment 2

[0099] Based on the image annotation method disclosed in Example 1, this embodiment discloses a target detection method based on graph matching. In this target detection method, the training sample images generated in Example 1 are mainly used to train and learn the image detection model.

[0100] In this embodiment, please refer to Figure 7 ,The target detection method based on graph matching includes steps S310-S330, which are described as follows.

[0101] Step S310, obtaining an image to be detected, wherein the image to be detected includes at least one detection target.

[0102] It should be noted that the image to be detected is similar to the image to be annotated. The detection target in the image is also the imaging state of a certain type of object. If various types of bolts are used as detection targets, the image to be detected needs to be automatically detected to find the bolts in the image and confirm the specific type of the bolts.

[0103] Step S320, detecting the image to be detected according to a preset image detection model, and obtaining node information of each node on the detection target and template information of the detection target.

[0104] In this embodiment, the image detection model is obtained by training a network model using the training sample images obtained in the above embodiment 1. For example, a convolutional neural network (CNN) for image detection is set as the network model, and the convolutional neural network is trained using one or more labeled training sample images, so that the convolutional neural network learns the image features of the target function, and the image detection model is formed by training by updating the network parameters.

[0105] Step S330, output the node information of each node on the detection target and the template information of the detection target.

[0106] It should be noted that since the nodes on the detection target represent the surface features of the detection target, the node information of each node includes semantic information, position information and scale information. The semantic information is the confidence of the represented image feature in each category, which is usually represented by one-hot encoding. The position information is the coordinates of the center point of the represented image feature on the image (including x-axis coordinates and y-axis coordinates), and the scale information is the size occupied by the represented image feature on the image (such as the number of pixels in length and width).

[0107] It should be noted that the template information may include the posture change of the detection target in terms of angle, area, and scale. It can be understood that the posture change refers to the change in the posture of the detection target in the candidate template relative to the posture of the standard part in the standard template, where the angle change includes the rotation change angle and shear change angle of the detection target, the area change includes the area change distance of the detection target, and the scale change refers to the zoom change multiple.

[0108] It can be understood that after obtaining the node information of each node on the detection target, the position of the detection target in the image to be detected and the specific type of the detection target are confirmed through the node information.

[0109] Those skilled in the art will appreciate that in this embodiment, an image detection model is obtained by training with labeled training sample images, so as to facilitate the use of the image training model to detect the detection target in the image to be detected. Among them, the use of multiple nodes for target recognition not only takes into account the similarity of image feature information, but also takes into account the relative position relationship between nodes, so as to obtain the azimuth change of the detection target through a multi-node method. In addition, the node method is used to comprehensively utilize the effective feature information of the detection target, and the stability of the image detection model is improved by eliminating useless information. The target detection method in this embodiment can reduce the operation time required for target recognition, and can also avoid the occurrence of false detection by eliminating redundant features, thereby improving the accuracy of target detection.

[0110] Those skilled in the art can understand that in the target detection method, since a network model is trained using labeled training sample images to obtain an image detection model, the image detection model has high stability and accuracy, and can accurately output the node information of each node on the detection target. In this embodiment, the posture problem is solved based on a preset standard template and a graph search technology, and the category problem is solved using a deep learning target detection technology, thereby decomposing the graph matching problem into two problems for solution, and then converting it into a graph matching algorithm for solution, which helps to improve the stability of graph matching. In addition, in the process of node search and posture matching, due to the use of a hierarchical matching strategy and template pose affine transformation, it is easy to obtain the node with the highest score as the final matching posture, thereby reducing the complexity of the template connection relationship and improving the algorithm operation efficiency.

[0111] Embodiment 3

[0112] On the basis of the image detection method disclosed in the second embodiment, this embodiment discloses a target detection device, which mainly uses the provided image detection method to perform detection processing on the image to be detected.

[0113] Please refer to Figure 8, the target detection device 4 in this embodiment includes an image acquisition component 41, a processor 42 and a display 43, which are described below respectively.

[0114] The image acquisition component 41 can be a camera, a video camera or other device with an image acquisition function, so the image acquisition component 41 can be used to acquire an image of at least one detection target in the scene and obtain an image to be detected. If the image acquisition component 41 is set on an industrial assembly line or a robot workbench, the detection target will be a product component on the assembly line, a tool on the workbench, etc.

[0115] The processor 42 is connected to the image acquisition component 41, which can be a chip or integrated circuit such as a CPU, an MCU, a programmable logic device, etc., and has the logic processing capability of data. Here, the processor 42 is used to obtain the node information of each node on the detection target according to the target detection method in the second embodiment, and / or obtain the identification type of the detection target.

[0116] If the detection target is a bolt, then the node information of each node on the detection target may include semantic information, position information and scale information of some image features on the bolt; the recognition type of the detection target is the bolt.

[0117] The display 43 is connected to the processor 42 and is used to display the node information of each node on the image to be detected and the detection target and / or the identification type of the detection target, so that the user can timely understand the detection result of the detection target in the image to be detected through the display 43.

[0118] In a specific embodiment, see Fig. 9 The processor 42 includes a training module 421 and a detection module 422, which are described as follows.

[0119] The training module 421 is used to train a network model using one or more training sample images, and obtain an image detection model by updating network parameters. The training sample images here are training sample images obtained by the image annotation method of the first embodiment.

[0120] The detection module 422 is connected to the training module 421, and is used to input the image to be detected into the image detection model and perform detection processing, and output the node information of each node on the detection target and / or the recognition type of the detection target.

[0121] In a specific embodiment, see Figure 8 and Fig.10 The processor 42 can realize the processing functions of multiple working stages, such as the processing work of the construction stage for the standard template, the processing work of the annotation stage for the image to be annotated, the processing work of the training stage for the image detection model, and the processing work of the detection stage for the image to be detected.

[0122] In the construction phase, the processor 42 obtains the standard part image of the standard part corresponding to the detection object, and sets a plurality of nodes on the standard part image for characterizing the surface features of the standard part. The processor 42 configures the search range of the standard part corresponding to the detection target on the standard part image according to the parameters input by the user interaction, and the search range includes the search angle, shear angle, search area and search scale of the standard part. Then, the processor 42 calculates the relative position between every two nodes on the standard part, obtains the edge information of every two nodes, and generates the graph structure of the standard part using every two nodes and the corresponding edge information. Next, the processor 42 constructs a standard template based on the search range of the standard part image and the graph structure of the standard part. It can be understood that the constructed standard template will play a role in the image annotation phase.

[0123] In the annotation stage, the processor 42 initially annotates each image to be annotated to obtain the first annotation information of the detection target, and the first annotation information includes the relative position information and search range information of several nodes on the detection target. Then, the processor 42 determines the search range of each of the several nodes according to the standard template obtained in the construction stage, searches for other nodes on the detection target within the search range of each node, and uses other nodes to form a candidate template, thereby performing a speculative analysis of the missing nodes on the candidate template according to the standard template to obtain the second annotation information of the detection target, and the annotated second annotation information includes the semantic information, position information and scale information of the missing nodes on the detection target. Next, the processor 42 generates a training sample image using the first annotation information and the second annotation information. It can be understood that the generated one or more training sample images are used to participate in the training of the network model in the training stage.

[0124] In the training phase, the processor 42 sets a convolutional neural network (CNN) for image detection as a network model, and trains the convolutional neural network using one or more labeled training sample images, so that the convolutional neural network learns the image features of the target function, and trains to form an image detection model by updating network parameters. It can be understood that since the image detection model has the function of detecting and identifying the detection target, the image detection model is used to play a role in the detection phase.

[0125] In the detection phase, the processor 42 first obtains the image to be detected from the image acquisition component 41, and then detects the image to be detected according to the image detection model obtained in the training phase, thereby obtaining the node information and template information of each node on the detection target, where the node information includes semantic information, position information and scale information, and the template information includes the posture change amount of the detection target in terms of angle, area and scale. It can be understood that if the node information and template information of the detection target are known, it can be confirmed that although the detection target has a posture difference with the standard part in the standard template, the two still belong to the same type, thereby determining the recognition type of the detection target.

[0126] Embodiment 4:

[0127] Please refer to Fig.11 In this embodiment, a target detection device is disclosed. The target detection device 5 includes a memory 51 and a processor 52, which are described below respectively.

[0128] The memory 51 is used to store programs. It can be understood that the stored programs may include program codes corresponding to steps S110-S140 in the first embodiment, and may also include program codes corresponding to steps S310-S330 in the second embodiment.

[0129] The processor 52 is connected to the memory 51 and is used to execute the program stored in the memory 51 to implement the image annotation method disclosed in the first embodiment, or to implement the target detection method disclosed in the second embodiment.

[0130] It should be noted that the specific functions implemented by the processor 52 may refer to the first and second embodiments, and may also refer to the processor 42 in the third embodiment, which will not be described in detail here.

[0131] Those skilled in the art will appreciate that all or part of the functions of the various methods in the above-mentioned embodiments can be implemented by hardware or by computer programs. When all or part of the functions in the above-mentioned embodiments are implemented by computer programs, the program can be stored in a computer-readable storage medium, and the storage medium can include: read-only memory, random access memory, disk, optical disk, hard disk, etc., and the program is executed by a computer to implement the above-mentioned functions. For example, the program is stored in the memory of the device, and when the program in the memory is executed by the processor, all or part of the above-mentioned functions can be implemented. In addition, when all or part of the functions in the above-mentioned embodiments are implemented by computer programs, the program can also be stored in a storage medium such as a server, another computer, disk, optical disk, flash disk or mobile hard disk, and can be downloaded or copied and saved in the memory of the local device, or the system of the local device is updated, and when the program in the memory is executed by the processor, all or part of the functions in the above-mentioned embodiments can be implemented.

[0132] The above specific examples are used to illustrate the present invention, which is only used to help understand the present invention and is not intended to limit the present invention. For those skilled in the art, according to the concept of the present invention, some simple deductions, modifications or substitutions can be made.

Claims

1. An image annotation method, characterized in that: include: Get the image to be annotated; The image to be labeled includes at least one detection target; Performing initial annotation on the image to be annotated to obtain first annotation information of the detection target; The first annotation information includes relative position information and search range information of a plurality of nodes on the detection target, wherein the nodes are used to characterize surface features of the detection target; Determine the search range of each of the plurality of nodes according to a preset standard template, search for other nodes on the detection target within the search range of each node, use the other nodes to form a candidate template, perform inference analysis on the missing nodes of the candidate template according to the standard template, and obtain second annotation information of the detection target; wherein, determine the relative positions of the nodes on the standard parts in the standard template according to the set of edge information between the nodes on the standard parts in the standard template, infer the nodes with similar position relationships in the set of candidate templates through template pose affine transformation, so as to obtain the missing nodes on the detection target, the standard parts refer to the standard reference objects of the detection target, and the second annotation information includes the semantic information, position information and scale information of the missing nodes on the detection target; A training sample image is generated using the first labeling information and the second labeling information.

2. The image annotation method according to claim 1, characterized in that: The initial annotation of the image to be annotated to obtain first annotation information of the detection target includes: Selecting a number of nodes on the detection target; The relative position information and search range information of the plurality of nodes are generated by annotation, thereby forming the first annotation information.

3. The image annotation method according to claim 1, characterized in that: The search range of each node includes the search angle, search area, clipping range and search scale of the node.

4. The image annotation method according to claim 1, characterized in that: The construction process of the standard template includes: Acquire a standard part image of the standard part corresponding to the detection target, and set a plurality of nodes for characterizing surface features of the standard part on the standard part image; Configuring a search range of the standard part corresponding to the detection target on the standard part image; the search range on the standard part image includes a search angle, a shear angle, a search area and a search scale of the standard part; Calculating the relative position between every two nodes on the standard part to obtain the edge information of every two nodes; generating the graph structure of the standard part using every two nodes and the corresponding edge information; The standard template is obtained according to the search range of the standard part image and the graph structure of the standard part.

5. The image annotation method according to claim 1, characterized in that: The performing the inference analysis of the missing nodes on the candidate template according to the standard template to obtain the second annotation information of the missing nodes on the detection target includes: Denote the standard template by G = (V1, A), and t ={(V t ,B)} represents the set of candidate templates; where V1 is the set of nodes on the standard part, A is the set of edge information between nodes on the standard part, V t is the set of nodes on the detection target corresponding to the t-th candidate template, and B is the set of edge information between the nodes on the detection target corresponding to the t-th candidate template; The relative positions of the nodes are determined according to the set A in G = (V1, A), and the template pose affine transformation is used to infer H = (V t ,B) nodes with similar position relationships, thereby obtaining the missing nodes on the detection target; Generate semantic information, position information and scale information of the missing nodes, build the connection relationship between the edges of the nodes according to the labeled nodes and the missing nodes, obtain the candidate template corresponding to the highest confidence, and thus form the second labeling information of the missing nodes.

6. The image annotation method according to claim 5, characterized in that: The relative positions of the nodes are determined according to the set A in G = (V1, A), and H is inferred by the template pose affine transformation. t ={(V t ,B)}, thereby obtaining the missing nodes on the detection target, including: The relative positions of the nodes in the set A in G = (V1, A) are determined by H t ={(V t ,B)}, the relative positions of the nodes determined by the set B are compared to infer the similarity relationship of the node positions, which can be expressed as Where n is the number of nodes in set B, i is the index number of the node, k is the number of instances of the i-th node in the image to be labeled, δ is the error and is expressed as Wherein, M represents the standard part image, E represents the image to be annotated, j is the index number, and ψ is the back-projection error function; Determine when the sum of the reflection and projection errors is minimized The corresponding candidate template, where b ij is the similarity of the nodes in set B; Find the missing nodes on the detection target according to the determined score value of the candidate template; the score value is expressed as Among them, d i is a process parameter and satisfies d i =||p i -(Rq i +t)|| 2 , p i is the coordinate vector of the i-th node in the standard template, q i is the coordinate vector of the i-th node in the determined candidate template, R is the rotation transformation relationship of the coordinate vector, t is the translation transformation relationship of the coordinate vector, n and m represent the total number of nodes on the detection target and the number of marked nodes respectively.

7. A target detection method based on graph matching, characterized in that: include: Acquire an image to be detected; the image to be detected includes at least one detection target; Detecting the image to be detected according to a preset image detection model to obtain node information of each node on the detection target and template information of the detection target; The image detection model is obtained by training a network model using the training sample images obtained by any method in claims 1-6; Output the node information and the template information; the node information includes semantic information, position information and scale information of the corresponding node, and the template information includes the posture change amount of the detection target in terms of angle, area and scale.

8. A target detection device, characterized in that: include: An image acquisition component, used to acquire an image of at least one detection target in a scene and obtain an image to be detected; A processor, connected to the image acquisition component, for obtaining node information of each node on the detection target, template information of the detection target, and / or obtaining an identification type of the detection target according to the target detection method described in claim 7; A display is connected to the processor and is used to display the image to be detected, the node information, the template information and / or the recognition type.

9. The target detection device according to claim 8, characterized in that: The processor includes a training module and a detection module; The training module is used to train a network model using one or more training sample images, and obtain an image detection model by updating network parameters; the training sample images are training sample images obtained by any method in claims 1-6; The detection module is connected to the training module, and is used to input the image to be detected into the image detection model and perform detection processing, and output the node information of each node on the detection target, the template information of the detection target, and / or the recognition type of the detection target.

10. A computer-readable storage medium, characterized in that: The method comprises a program, wherein the program can be executed by a processor to implement the image annotation method as described in any one of claims 1 to 6, or to implement the target detection method as described in claim 7.

Images