Image detection method, device, equipment and storage medium

Abstract

The embodiments of the present application disclose an image detection method, apparatus, device and storage medium, wherein the method comprises: processing an image to be processed based on a first segmentation model to obtain an image to be detected corresponding to an object to be detected, wherein the image to be processed is acquired when the object to be detected is located in the scene in which it is located; wherein a first label for training the first segmentation model is determined based on the second segmentation model, wherein the second segmentation model is obtained by training a plurality of fused images carrying a second label, wherein the fused image is formed by fusing a sub-image corresponding to the detection object and sub-images corresponding to other scenes, wherein the sub-image corresponding to the detection object is determined based on the second label; and detecting the image to be detected to obtain a detection result corresponding to the object to be detected.

Description

Technical Field

[0001] The present application relates to computer processing technology, and in particular to an image detection method, device, equipment and storage medium. Background Art

[0002] In the related technology, when the inspection vehicle conducts safety inspection on the high-speed rail, the background in the tunnel is very complex and the background is easily confused with the parts and defects to be inspected. Therefore, it is difficult to detect the parts and defects from the collected images, resulting in a large number of false detections. In addition, the background is very complex, resulting in very large interference, so the application of anomaly detection algorithms is also very difficult. Summary of the invention

[0003] In view of this, embodiments of the present application provide an image detection method, apparatus, device, and storage medium.

[0004] In a first aspect, an embodiment of the present application provides an image detection method, the method comprising:

[0005] Processing the image to be processed based on the first segmentation model to obtain an image to be detected corresponding to the object to be detected;

[0006] In some embodiments, the image to be processed is acquired when the object to be detected is located in the scene where it is located.

[0007] In some embodiments, a first label used to train the first segmentation model is determined based on the second segmentation model, the second segmentation model is trained by multiple fused images carrying a second label, the fused image is formed by fusing a sub-image corresponding to the detection object and sub-images corresponding to other scenes, and the sub-image corresponding to the detection object is determined based on the second label.

[0008] The image to be detected is detected to obtain a detection result corresponding to the object to be detected.

[0009] In a second aspect, an embodiment of the present application provides an image detection device, the device comprising:

[0010] A first processing module, used for processing the image to be processed based on the first segmentation model to obtain an image to be detected corresponding to the object to be detected;

[0011] In some embodiments, the image to be processed is acquired when the object to be detected is located in the scene where it is located.

[0012] In some embodiments, the first label used to train the first segmentation model is determined based on the second segmentation model, the second segmentation model is trained by multiple fused images carrying masks, the fused image is formed by fusing a sub-image corresponding to the detection object and sub-images corresponding to other scenes, and the sub-image corresponding to the detection object is determined based on the mask.

[0013] The second processing module is used to detect the image to be detected and obtain the detection result corresponding to the object to be detected.

[0014] In a third aspect, an embodiment of the present application provides an electronic device, the electronic device comprising: a processor, a memory, and a communication bus;

[0015] The communication bus is used to realize the communication connection between the processor and the memory;

[0016] The processor is used to execute the program in the memory to implement the above-mentioned image detection method.

[0017] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the above-mentioned image detection method.

[0018] In the embodiments of the present application, first, the image to be processed is processed based on the first segmentation model to obtain the image to be detected corresponding to the object to be detected; wherein, the image to be processed is acquired when the object to be detected is located in the scene where it is located; the first label for training the first segmentation model is determined based on the second segmentation model, the second segmentation model is trained by multiple fused images carrying the second label, the fused image is formed by fusing the sub-image corresponding to the detection object and the sub-image corresponding to other scenes, and the sub-image corresponding to the detection object is determined based on the second label; secondly, the image to be detected is detected to obtain the detection result corresponding to the object to be detected; in this way, using the first segmentation model trained based on a large number of fused images carrying the second label to process the image to be processed can save manual labeling costs, accurately detect components from complex backgrounds, improve segmentation accuracy, and improve detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A schematic diagram of an implementation flow of an image detection method provided in an embodiment of the present application;

[0020] Figure 2 A schematic diagram of an implementation flow of an image detection method provided in an embodiment of the present application;

[0021] Figure 3 A schematic diagram of an implementation flow of an image detection method provided in an embodiment of the present application;

[0022] Figure 4 A schematic diagram of an implementation flow of an image detection method provided in an embodiment of the present application;

[0023] Figure 5A A schematic diagram of an implementation flow of an image detection method provided in an embodiment of the present application;

[0024] Figure 5B A schematic diagram of an implementation flow of an image detection method provided in an embodiment of the present application;

[0025] Figure 6 A schematic diagram of an implementation flow of an image detection method provided in an embodiment of the present application;

[0026] Figure 7 A schematic diagram of an implementation flow of an image detection method provided in an embodiment of the present application;

[0027] Figure 8 A schematic diagram of an implementation flow of an image detection method provided in an embodiment of the present application;

[0028] Fig.9A A schematic diagram of the implementation flow of a defect detection method in a complex scenario provided in an embodiment of the present application;

[0029] Fig. 9B A schematic diagram of the implementation flow of a defect detection method in a complex scenario provided in an embodiment of the present application;

[0030] Fig.10 A schematic diagram of the structure of an image detection device provided in an embodiment of the present application;

[0031] Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] The technical solution of the present application is further described in detail below with reference to the accompanying drawings and embodiments.

[0033] Figure 1 A schematic diagram of the implementation flow of the state detection method provided in the embodiment of the present application, such as Figure 1 As shown, the method includes:

[0034] Step S101: Processing the image to be processed based on the first segmentation model to obtain the image to be detected corresponding to the object to be detected.

[0035] In some embodiments, the image to be processed is acquired when the object to be detected is located in the scene where it is located.

[0036] The object to be detected may be a component, such as a suspension string, a current-carrying ring, a load-bearing cable, etc. The object to be detected is located in the scene where it is located, which can be understood as the object to be detected is located in a real scene, such as inside a tunnel or outside a tunnel. At this time, the image to be processed may be an image including components and background collected in a real scene, such as an image including components and background collected inside a tunnel, and an image including components and background collected outside a tunnel.

[0037] In some embodiments, the image to be processed may include multiple objects to be detected; in one possible implementation, the image to be processed includes multiple components; in one possible implementation, the image to be processed includes multiple connectors; in one possible implementation, the image to be processed includes multiple components and connectors.

[0038] In some embodiments, the image to be detected may be a sub-image corresponding to the object to be detected; for example, if the object to be detected is a hanging string, the image to be detected is an image that only includes the hanging string but not the background; if the object to be detected is a fastener, the image to be detected is an image that only includes the fastener but not the background.

[0039] In a possible implementation, the image to be detected is an image obtained by segmenting the object to be detected from the image to be processed using a first segmentation model; in a possible implementation, the image to be processed includes multiple objects to be detected, and the images to be detected obtained by processing using the first segmentation model are multiple, wherein one object to be detected corresponds to one image to be detected; in a possible implementation, the image to be processed includes one object to be detected, and the image to be detected obtained by processing using the first segmentation model is one.

[0040] In some embodiments, a first label used to train the first segmentation model is determined based on the second segmentation model, the second segmentation model is trained by multiple fused images carrying a second label, the fused image is formed by fusing a sub-image corresponding to the detection object and sub-images corresponding to other scenes, and the sub-image corresponding to the detection object is determined based on the second label.

[0041] In some embodiments, the first label can be obtained based on the label output by the second segmentation model; the second label can be a mask, contour information, etc., which is a label of the sub-image corresponding to the detection object, and is generated in the process of determining the sub-image corresponding to the detection object. In a possible implementation, the second label is the label input by the first model training, and the first label is the label input by the second model training; in a possible implementation, the first label can be obtained by manually adjusting the label output by the second segmentation model, so that a more accurate first label can be obtained.

[0042] In some embodiments, the sub-images corresponding to other scenes may be background images; the sub-images corresponding to other scenes may be understood as complex background images, that is, when a component is located in the complex background image, the difficulty of detecting the component will increase, such as the background image in a tunnel; since the background in the tunnel is complex, the background and components are easily confused, and therefore, the difficulty of detecting the component increases.

[0043] In some embodiments, the sub-image corresponding to the detection object can be determined from an image with a simple background and including the detection object, that is, the difficulty of detecting the object to be detected from the image will be reduced, such as: the image outside the tunnel; because the background outside the tunnel is simple, the background and the detection object are not easily confused, therefore, the difficulty of detecting the detection object is reduced.

[0044] In some embodiments, the detection object and the sub-image corresponding to the detection object are used in the model training process, and the object to be detected and the image to be processed are used in the model application process.

[0045] In some embodiments, the sub-image corresponding to the detection object is determined based on the second label, that is, the second label is used to determine the sub-image corresponding to the detection object; in actual applications, after the sub-image corresponding to the detection object is determined based on the second label, the sub-image corresponding to the detection object carries the second label and is fused with sub-images corresponding to other scenes to obtain a fused image carrying the second label; in this way, the cost of manual labeling can be saved.

[0046] Step S102: Detect the image to be detected to obtain a detection result corresponding to the object to be detected.

[0047] In some embodiments, the detection result represents the safety hazard existing in the object to be detected; in one possible implementation, the object to be detected has no safety hazard; in one possible implementation, the object to be detected has one or more safety hazards; wherein, the multiple safety hazards may be the same or different.

[0048] In some embodiments, the detection can be performed on multiple images to be detected at the same time, or on one image to be detected. The purpose of the detection is to determine the safety of the object to be detected in the image to be detected. If the detection result shows that the object to be detected cannot be used safely, the maintenance personnel will perform corresponding maintenance for the specific safety hazards of the object to be detected.

[0049] In some embodiments, detection can be achieved through a fast target detection algorithm; in one possible implementation, a faster classification algorithm can be directly used as a feature extraction network for target detection; in one possible implementation, a model acceleration compression algorithm can be used to accelerate the target detection algorithm before performing target detection; in one possible implementation, the network structure can be redesigned, including feature extraction and detection networks.

[0050] In one possible implementation, detection can be achieved through the following process: first, feature extraction is performed on the image to be detected; second, a region is generated based on the extracted features, and the region can be a pre-selected box including the object to be detected; finally, the image corresponding to the above region is classified or positioned and regressed through a convolutional neural network to obtain the category information and position information of the object to be detected.

[0051] In one possible implementation, detection can be achieved through the following process: first, feature extraction is performed on the image to be detected; second, the extracted features are classified or positioned and regressed through a convolutional neural network to obtain category information and position information of the object to be detected.

[0052] In an embodiment of the present application, first, the image to be processed is processed based on the first segmentation model to obtain an image to be detected corresponding to the object to be detected, and the image to be processed is collected when the object to be detected is located in the scene where it is located; wherein, the first label used to train the first segmentation model is determined based on the second segmentation model, and the second segmentation model is trained by multiple fused images carrying the second label, and the fused image is formed by fusing a sub-image corresponding to the detection object and sub-images corresponding to other scenes, and the sub-image corresponding to the detection object is determined based on the second label; secondly, the image to be detected is detected to obtain a detection result corresponding to the object to be detected; in this way, using the first segmentation model trained based on a large number of fused images carrying the second label to process the image to be processed can save manual labeling costs, accurately detect components from complex backgrounds, improve segmentation accuracy, and improve detection accuracy.

[0053] In some embodiments, the first segmentation model is trained by multiple processed images carrying the first label, and the first label is obtained by predicting the processed image based on the second segmentation model.

[0054] In some embodiments, the processed image can be understood as a real image with a complex background during the model training process; the difference between the processed image and the image to be processed is that the image to be processed is the input image of the model application process, while the processed image is the verification image of the model training process. Correspondingly, the first label can be understood as a real label; in a possible implementation, the first label can be an accurate label obtained by manually adjusting the label output by the second segmentation model, for example: based on manual experience, the label with errors output by the second segmentation model is adjusted to make it more accurate.

[0055] Figure 2 A schematic diagram of the implementation flow of the state detection method provided in the embodiment of the present application, such as Figure 2 As shown, the training method of the first segmentation model includes:

[0056] Step S201: Performing model training based on the multiple fused images carrying the second label to obtain the second segmentation model.

[0057] Step S202: using the second segmentation model to predict multiple processed images to obtain multiple processed images carrying prediction labels.

[0058] In some embodiments, the processed image is used to verify the labeling effect of the second segmentation model on the real image with complex background, that is, the second segmentation model is used to predict the processed image (real image with complex background) and can obtain the predicted label of the real image with complex background; wherein the predicted label can be a mask, contour information, etc.

[0059] Step S203: for each processed image carrying the predicted label, obtaining an adjustment result for the predicted label to obtain a processed image carrying the first label.

[0060] In some embodiments, the adjustment of the predicted label is to adjust the label with errors output by the second segmentation model. In a possible implementation, the contour parameters of the second label may be adjusted, such as the smoothness of the contour.

[0061] Step S204: Perform model training based on multiple processed images carrying the first label to obtain the first segmentation model.

[0062] In some embodiments, the first segmentation model and the second segmentation model may be the same or different; in one possible implementation, the first segmentation model may be obtained by optimizing the second segmentation model, for example: the model is trained once using a plurality of fused images carrying the second label to obtain the second segmentation model; and then the model is trained a second time using a plurality of processed images carrying the second label to obtain the first segmentation model; in one possible implementation, the first segmentation model may be obtained by training a segmentation model different from the second segmentation model, for example: the model is trained using a plurality of fused images carrying the second label to obtain the second segmentation model; and then the model is trained using a plurality of processed images carrying the first label to obtain the first segmentation model; wherein the training models of the first segmentation model and the second segmentation model are different.

[0063] In an embodiment of the present application, first, model training is performed based on the multiple fused images carrying the second label to obtain the second segmentation model; secondly, the second segmentation model is used to detect multiple processed images to obtain multiple processed images carrying predicted labels; thirdly, for each processed image carrying the predicted label, an adjustment result for the predicted label is obtained to obtain a processed image carrying the first label; finally, model training is performed based on the multiple processed images carrying the first label to obtain the first segmentation model; in this way, the annotation cost of the real image can be reduced, and a more accurate model can be obtained.

[0064] Figure 3 A schematic diagram of the implementation flow of the state detection method provided in the embodiment of the present application, such as Figure 3 As shown, the method for obtaining the fused image includes:

[0065] Step S301: Acquire a detection image.

[0066] In some embodiments, the detection image includes the detection object; here, the detection image can be understood as an image used to extract a sub-image corresponding to the detection object, including the detection object and the background, and the background is simple, such as: the image outside the tunnel; because the background outside the tunnel is simple, the background and the detection object are not easily confused, so the difficulty of detecting the detection object is reduced, and it is easy to obtain the sub-image corresponding to the detection object, thereby reducing the difficulty of sample acquisition.

[0067] Step S302: Identify the detection image to determine the position information of the detection object in the detection image.

[0068] In some embodiments, the detection image can be detected by a target detection algorithm to determine the position information of the detection object in the detection image. Here, the position information can be understood as a pre-selected box including the detection object.

[0069] Step S303: Based on the position information of the detection object and the detection image, generate a sub-image corresponding to the second label and the detection object.

[0070] In some embodiments, the second label is used to determine the sub-image corresponding to the detection object, and is also output simultaneously with the sub-image corresponding to the detection object as a label of the detection object; in one possible implementation, the second label may be a mask; in one possible implementation, the second label may be contour information; in one possible implementation, step S303 may be implemented by the following process: first, based on the position information of the detection object, copy the part of the detection image where the detection object is located; secondly, based on the part of the detection image, determine the mask or contour information of the detection object; finally, based on the mask or contour information of the detection object, cut out the sub-image corresponding to the detection object from the detection image.

[0071] Step S304: synthesize the sub-image corresponding to the other scene and the sub-image corresponding to the detection object into the fused image, wherein the fused image carries the second label.

[0072] In some embodiments, the sub-images corresponding to other scenes can be understood as complex background images used for model training; in a possible implementation, the collected complex background images of other scenes can be directly used as sub-images corresponding to other scenes; in a possible implementation, the collected complex background images of other scenes can be downsampled to obtain sub-images corresponding to other scenes; here, the size of the sub-images corresponding to other scenes is greater than or equal to the size of the sub-images corresponding to the detection object. In addition, since the second label is output simultaneously with the sub-image corresponding to the detection object, the fused image synthesized by the sub-images corresponding to other scenes and the sub-images corresponding to the detection object carries the second label.

[0073] In some embodiments, a small number of sub-images corresponding to other scenes and sub-images corresponding to the detection object can be synthesized by image enhancement to obtain a large number of fused images; in a possible implementation method, the sub-image corresponding to the detection object is spatially transformed (for example, rotated, scaled, mirrored, translated, etc.), and then synthesized with sub-images corresponding to other scenes to obtain a large number of fused images; in a possible implementation method, the sub-image corresponding to other scenes is spatially transformed (for example, rotated, scaled, mirrored, translated, etc.), and then synthesized with the sub-image corresponding to the detection object to obtain a large number of fused images.

[0074] In an embodiment of the present application, first, a detection image is acquired; wherein, the detection image includes the detection object; secondly, the detection image is identified to determine the position information of the detection object in the detection image; thirdly, based on the position information of the detection object and the detection image, the second label and a sub-image corresponding to the detection object are generated; then, the sub-images corresponding to the other scenes and the sub-images corresponding to the detection object are synthesized into the fused image, and the fused image carries the second label; in this way, a large number of fused images carrying the second label can be obtained based on a small number of sub-images corresponding to other scenes and sub-images corresponding to the detection object, and the cost of manual labeling is saved.

[0075] Figure 4 A schematic diagram of the implementation flow of the state detection method provided in the embodiment of the present application, such as Figure 4 As shown, step S303 includes:

[0076] Step S401: Based on the position information of the detection object, binarize the detection object to obtain a second label of the detection object.

[0077] In some embodiments, the second label may be a mask, contour information, etc. of the detection object.

[0078] In the case where the second label is a mask of the detection object, the second label can be determined by the following process: first, based on the position information of the detection object, the detection object is copied to obtain a copied image; secondly, the segmentation threshold is determined; thirdly, the pixel value of each pixel in the copied image is calculated; then, the pixel value greater than or equal to the segmentation threshold pixel is used as the second label.

[0079] In the case where the second tag is the contour information of the detection object, the second tag can be implemented through the following process: first, edge detection is performed on the detection object; second, the contour information of the detection object is extracted based on the edge detection result.

[0080] Step S402: Process the detection image using the second label of the detection object to obtain a sub-image corresponding to the detection object.

[0081] In some embodiments, the area corresponding to the second label in the detection image can be determined as the sub-image corresponding to the detection object; in a possible implementation, the sub-image corresponding to the detection object can be obtained by multiplying the second label of the detection object with the detection image; the principle is: the pixels with a pixel value of 0 in the second label of the detection object are multiplied with the pixels in the detection image, and the pixel values ​​of some pixels in the detection image are changed to 0; the pixels with a pixel value of 1 in the second label of the detection object are multiplied with the pixels in the detection image, and the pixel values ​​of some pixels in the detection image remain unchanged; in this way, the image composed of pixels with unchanged pixel values ​​in the detection image is determined as the sub-image corresponding to the detection object. In a possible implementation, the area corresponding to the contour information of the detection object can be determined as the sub-image corresponding to the detection object.

[0082] In an embodiment of the present application, first, based on the position information of the detection object, the detection object is binarized to obtain a second label of the detection object; secondly, the detection image is processed using the second label of the detection object to obtain a sub-image corresponding to the detection object; in this way, the sub-image of the detection object can be determined from the detection image based on the second label of the detection object.

[0083] Figure 5A A schematic diagram of the implementation flow of the state detection method provided in the embodiment of the present application, such as Figure 5A As shown, step S102 includes:

[0084] Step S501: Detect the image to be detected from at least two detection dimensions to obtain a detection result of the object to be detected in each of the detection dimensions.

[0085] In some embodiments, the detection result includes location information and category information. Here, there is a corresponding relationship between the detection dimension, the object to be detected, and the detection result; the objects to be detected under different detection dimensions are different, and the detection results are also different.

[0086] In a possible implementation, the detection dimension is a component detection dimension, the corresponding object to be detected is the component in the image to be detected, and the detection result is the position information and category information of the component, for example: under the component detection dimension, the object to be detected is component A, and the detection result is position 1 and hanging string; in a possible implementation, the detection dimension is a defect detection dimension, the corresponding object to be detected is the defect in the image to be detected, and the detection result is the position information and category information of the defect, for example: the object to be detected is defect a, and the detection result is position 2 and broken.

[0087] In some embodiments, two target detection algorithms may be used to detect components and defects respectively, for example: a first target detection algorithm is used to detect components, and a second target detection algorithm is used to detect defects. In a possible implementation, components and defects in the image to be detected may be detected at the same time; in a possible implementation, one of the components and defects in the image to be detected may be detected, and then the remaining one may be detected.

[0088] Step S502: Associating the objects to be detected in the at least two detection dimensions based on the position information to obtain association results corresponding to the associated objects.

[0089] In some embodiments, the association result includes category information corresponding to the associated object in each of the detection dimensions. For example, component A and defect a are determined as associated objects, the category information of component A is hanging string, and the category information of defect a is damaged, then hanging string-damaged is used as the association result between component A and defect a.

[0090] In some embodiments, association refers to matching the component and the defect according to the location information of the component and the location information of the defect to form a one-to-one correspondence between the component and the defect, such as Figure 5B As shown, for example: the position information of component A and the position information of defect a represent that there is an association relationship between the two, so component A and defect a are associated to obtain a group of associated objects; the position information of component A and the position information of defect a and defect b represent that there is an association relationship between the two, so component A and defect a are associated, and component A and defect b are associated to obtain two groups of associated objects.

[0091] Step S503: verifying the association result corresponding to the associated object based on the category association information, and obtaining the target detection result corresponding to the object to be detected.

[0092] In some embodiments, the category association information is an association relationship between category information of at least two detection dimensions; here, the category association information can be understood as a one-to-one correspondence between category information of at least two preset detection dimensions; for example: objectively, the defects that may occur on component A are defect a and defect b, and the defect that may occur on component B is defect c, then a one-to-one correspondence is established between component A-defect a, component A-defect b, and component B-defect c.

[0093] In some embodiments, the category association information can be regarded as a database, and the association result corresponding to the associated object is verified based on the category association information, that is, the association result is compared with the category association information in the database. If it exists in the database, the association result is output; if it does not exist in the database, the association result is discarded; for example: when there is an association result component A-defect a, the association result component A-defect a is compared with the category association information component A-defect a, component A-defect b, component B-defect c in the database, and there is category association information consistent with the association result in the database, then the association result component A-defect a is output; when there is an association result component A-defect c, the association result component A-defect c is compared with the category association information component A-defect a, component A-defect b, component B-defect c in the database, and there is no category association information consistent with the association result component A-defect c in the database, then the association result component A-defect c is discarded.

[0094] In an embodiment of the present application, first, the image to be detected is detected from at least two detection dimensions to obtain the detection results of the object to be detected in each detection dimension; secondly, the objects to be detected in the at least two detection dimensions are associated based on the position information to obtain the association results corresponding to the associated objects; finally, the association results corresponding to the associated objects are verified based on the category association information to obtain the target detection results corresponding to the object to be detected; in this way, the association results can be verified based on the preset category association information, thereby reducing false detections and improving the detection accuracy.

[0095] Figure 6 A schematic diagram of the implementation flow of the state detection method provided in the embodiment of the present application, such as Figure 6 As shown, step S502 includes:

[0096] Step S601: Based on the position information of the object to be detected in the at least two detection dimensions, the object to be detected in the at least two detection dimensions with associated positions is determined as the associated object.

[0097] In some embodiments, position association can be that the position information of the objects to be inspected in two detection dimensions overlaps, for example: if the position information of component A overlaps with the position information of defect a, then component A and defect a are associated to obtain a group of associated objects; if the position information of component A overlaps with the position information of defect a and defect b, then component A and defect a are associated to obtain a group of associated objects.

[0098] In some embodiments, the location information may be an area, and by comparing whether the areas of the objects to be detected in different detection dimensions overlap, the objects to be detected in different detection dimensions with overlapping areas are associated to obtain associated objects; here, a group of associated objects includes a one-to-one correspondence. In a possible implementation, the location information may be represented by coordinates.

[0099] Step S602: establishing a corresponding relationship between the category information of the associated objects in each of the detection dimensions to obtain the association result.

[0100] In some embodiments, the association result includes the category information corresponding to the associated object in each of the detection dimensions; for example: component A and defect a are associated objects, the category information of component A is hanging string, and the category information of defect a is damaged, then the association result is hanging string-broken; component A and defect a, component A and defect b are associated objects, the category information of component A is hanging string, the category information of defect a is damaged, and the category information of defect b is broken, then the association results are hanging string-broken, hanging string-broken.

[0101] In an embodiment of the present application, firstly, based on the position information of the object to be detected in the at least two detection dimensions, the object to be detected in the at least two detection dimensions with position association is determined as the associated object; secondly, a correspondence between the category information of the associated object in each of the detection dimensions is established to obtain the association result; in this way, based on the position information of the object to be detected in different detection dimensions, the category information of the object to be detected in different dimensions can be associated to determine what kind of defect exists on what kind of component.

[0102] In some embodiments, the category association information includes an association relationship between each of the preset objects in a plurality of preset objects, and the association relationship includes category information corresponding to the preset object in each of the detection dimensions. Figure 7 A schematic diagram of the implementation flow of the state detection method provided in the embodiment of the present application, such as Figure 7 As shown, step S503 includes:

[0103] Step S701: comparing the correspondence between the category information of the associated objects in each of the detection dimensions in each of the association results with the category information corresponding to the plurality of preset objects in each of the detection dimensions.

[0104] In some embodiments, the preset objects can be understood as components and defects that can be associated under objective circumstances. For example, under objective circumstances, the possible defects of component A are defect a, defect b, and defect c, then the preset objects are component A-defect a, component A-defect b, and component A-defect c.

[0105] In some embodiments, the category information corresponding to the preset object in each detection dimension can be understood as the correspondence between the category information of the preset objects. For example, the category information of component A is hanging string, the category information of defect a is broken, the category information of defect b is bulge, and the category information of defect c is damaged. Then the category information corresponding to the preset object in each detection dimension is: hanging string-broken, hanging string-bulge, hanging string-damaged.

[0106] In some embodiments, when a component is associated with one defect, the inspection result of the component includes a one-to-one correspondence, for example: if the hanging string of component A is associated with defect a, which is broken, then the inspection result of the hanging string of component A is hanging string-broken; when a component is associated with multiple defects, the inspection result of the component includes multiple one-to-one correspondences, for example: if the hanging string of component A is associated with defect a, which is broken, and defect b, which is bulging, then the inspection results of the hanging string of component A are hanging string-broken and hanging string-broken.

[0107] In some embodiments, the preset object can be regarded as a database, and the comparison is to compare the association result with the category information corresponding to the preset object in each detection dimension in the database. For example: for the association results of hanging string-broken and hanging string-bulged, they all exist in the database hanging string-broken, hanging string-bulged, and hanging string-damaged; for the association result hanging string-etched, it does not exist in the database hanging string-broken, hanging string-bulged, and hanging string-damaged.

[0108] Step S702: When the comparison results are consistent, the detection result of the object to be detected corresponding to the category information of at least two detection dimensions in the associated results is used as the target detection result corresponding to the object to be detected.

[0109] In some embodiments, when there is a preset object consistent with the association result in the data, the association result is output and used as the final detection result; when there is no preset object consistent with the detection result in the data, the association result is discarded. For example: for the association results hanging string-broken and hanging string-bulged, if they exist in the database hanging string-broken, hanging string-bulged, and hanging string-damaged, the association results hanging string-broken and hanging string-bulged are output as the final detection result; for the association result hanging string-etched, if they do not exist in the database hanging string-broken, hanging string-bulged, and hanging string-damaged, the association result is discarded.

[0110] In some embodiments, when multiple images to be detected are detected at the same time, the output is the detection results of multiple components; for example: two images to be detected are detected at the same time, where the object to be detected in one image to be detected is component A, and the object to be detected in the other image to be detected is component B, the detection result of component A is suspension string-broken, suspension string-bulged, and the detection result of component B is load-bearing cable-broken, then the detection result of component A and the detection result of component B are output as the final detection result.

[0111] In an embodiment of the present application, first, the correspondence between the category information of the associated objects in each of the detection dimensions in each of the association results is compared with the category information corresponding to the multiple preset objects in each of the detection dimensions; secondly, when the comparison results are consistent, the detection results of the object to be detected corresponding to the category information of at least two detection dimensions in the associated results are used as the target detection results corresponding to the object to be detected; in this way, the detection results can be screened to obtain more accurate detection results, thereby improving the detection accuracy.

[0112] In some embodiments, the at least two detection dimensions include a defect detection dimension and a component detection dimension, Figure 8 A schematic diagram of the implementation flow of the state detection method provided in the embodiment of the present application, such as Figure 8 As shown, the method comprises the following steps:

[0113] Step S801: Detect the image to be detected from the defect detection dimension to obtain a defect detection result of the defect object in the defect detection dimension; the defect detection result includes the position information of the defect object and the defect category of the defect object; detect the image to be detected from the component detection dimension to obtain a component detection result of the component object in the component detection dimension; the component detection result includes the position information of the component object and the component category of the component object.

[0114] Step S802: Based on the location information of the defect object and the location information of the component object, the defect object and the component object are associated to obtain an association result corresponding to the associated object; the association result includes an association relationship between the defect category of the defect object and the component category of the component object.

[0115] Step S803: verifying the association relationship between the defect category of the defect object and the component category of the component object based on preset category association information to obtain a target detection result corresponding to the object to be detected; the category association relationship is the association relationship between the defect category and the component category.

[0116] In an embodiment of the present application, the image to be inspected is inspected from the defect detection dimension and the component detection dimension, and the defect detection results and the component detection results are associated and verified to obtain the final target detection result; in this way, false detections can be reduced and detection accuracy can be improved.

[0117] Below, an exemplary application of an embodiment of the present application in an actual application scenario will be described, and the following description will be given by taking a defect detection method in a complex scenario as an example.

[0118] Fig.9A A schematic diagram of the implementation flow of a defect detection method in a complex scenario provided in an embodiment of the present application is shown in FIG. Fig.9A As shown, the method includes:

[0119] Step S901: Acquire an image outside the tunnel;

[0120] In some embodiments, the image outside the tunnel includes the component and the background, and the background is simple; that is, the component is clearly visible.

[0121] In some embodiments, the image outside the tunnel corresponds to the above-mentioned detection image.

[0122] Step S902: performing target detection on the image outside the tunnel and outputting the position information of the components;

[0123] In some embodiments, the position information of the component corresponds to the position information of the detection object described above.

[0124] Step S903: Binarizing the image outside the tunnel based on the position information of the component to obtain a mask of the component;

[0125] In some embodiments, the mask of the component corresponds to the second label mentioned above.

[0126] Step S904: using the mask of the component to process the image outside the tunnel, and obtain a sub-image corresponding to the component;

[0127] In some embodiments, the sub-image corresponding to the component corresponds to the sub-image corresponding to the detection object.

[0128] Step S905: synthesize the sub-image corresponding to the component and the image in the tunnel to obtain a fused image;

[0129] In some embodiments, the image in the tunnel includes a background, and the background is complex.

[0130] In some embodiments, the image in the tunnel corresponds to the sub-image corresponding to the other scenes mentioned above.

[0131] Step S906: using the fused image to perform model training to obtain a second segmentation model;

[0132] Step S907: using the second segmentation model to predict the real image to obtain a real image carrying a predicted label;

[0133] Step S908: Obtain the adjustment result for the prediction, and obtain the real image carrying the real label;

[0134] In some embodiments, the real label corresponds to the first label; the real image corresponds to the processed image.

[0135] Step S909: Perform model training using real images with real labels to obtain a first segmentation model;

[0136] In some embodiments, the actual operations of the above steps S901 to S909 are as follows: Fig. 9B As shown, 91 represents the image outside the tunnel obtained in step S901, 92 represents the mask of the component obtained in step S903, 93 represents the sub-image corresponding to the component obtained in step S904, 94 and 95 can both represent the image inside the tunnel in step S905, 96 represents the fused image obtained in step S905, 97 represents the second segmentation model obtained in step S906, 98 represents the real image with the predicted label obtained in step S907, and 99 represents the first segmentation model obtained in step S909; wherein 94 and 95 can both represent the image inside the tunnel in step S905. In practice, the following two methods of obtaining the fused image can be included:

[0137] Case 1: The acquired tunnel image 94 is directly synthesized with the sub-image corresponding to the component to obtain a fused image;

[0138] Case 2: The acquired tunnel image 94 is sampled to obtain an image 95, which is then synthesized with the sub-image corresponding to the component to obtain a fused image.

[0139] Step S910: segmenting the image to be processed using the first segmentation model to obtain a component to be detected;

[0140] In some embodiments, the component to be detected corresponds to the above-mentioned object to be detected.

[0141] Step S911: inspecting the component to be inspected from a defect inspection dimension and a component inspection dimension to obtain a defect inspection result and a component inspection result;

[0142] Step S912: establishing a correspondence between defects and components having overlapping position information according to the position information in the defect detection results and the component detection results;

[0143] Step S913: comparing the correspondence between defects and components with the preset correspondence between defects and components;

[0144] In some embodiments, the correspondence between the preset defects and the components corresponds to the above-mentioned category association information.

[0145] Step S914: When the correspondence between the defect and the component is consistent with the preset correspondence between the defect and the component, the correspondence between the defect and the component is determined as the final inspection result.

[0146] The present application embodiment provides an image detection device, Fig.10 A schematic diagram of the structure of a fall detection device provided in an embodiment of the present application is shown in FIG. Fig.10 As shown, the image detection device 1000 includes:

[0147] The first processing module 1001 is used to process the image to be processed based on the first segmentation model to obtain the image to be detected corresponding to the object to be detected.

[0148] In some embodiments, the image to be processed is acquired when the object to be detected is in the scene in which it is located; the first label used to train the first segmentation model is determined based on the second segmentation model, the second segmentation model is trained by multiple fused images carrying the second label, the fused image is formed by fusing a sub-image corresponding to the detection object and sub-images corresponding to other scenes, and the sub-image corresponding to the detection object is determined based on the second label.

[0149] The second processing module 1002 is used to detect the image to be detected and obtain a detection result corresponding to the object to be detected.

[0150] In some embodiments, the first segmentation model is trained by multiple processed images carrying the first label, and the first label is obtained by predicting the processed image based on the second segmentation model.

[0151] In some embodiments, the first processing module 1001 includes:

[0152] A first input module, configured to perform model training based on the plurality of fused images carrying the second label to obtain the second segmentation model;

[0153] A first processing submodule, configured to predict a plurality of processed images using the second segmentation model to obtain a plurality of processed images carrying prediction labels;

[0154] A first acquisition submodule, configured to acquire, for each processed image carrying the predicted label, an adjustment result for the predicted label to obtain a processed image carrying the first label;

[0155] The second input module is used to perform model training based on multiple processed images carrying the first label to obtain the first segmentation model.

[0156] In some embodiments, the first processing module 1001 further includes:

[0157] The second acquisition submodule is used to acquire the detection image;

[0158] In some embodiments, the detection image includes the detection object;

[0159] A second processing submodule is used to identify the detection image and determine the position information of the detection object in the detection image;

[0160] A third processing submodule, configured to generate a sub-image corresponding to the second label and the detection object based on the position information of the detection object and the detection image;

[0161] A fourth processing submodule, configured to synthesize the sub-image corresponding to the other scenes and the sub-image corresponding to the detection object into the fused image;

[0162] In some embodiments, the fused image carries the second label.

[0163] In some embodiments, the third processing submodule includes:

[0164] A fifth processing submodule, configured to perform binarization processing on the detection object based on the position information of the detection object to obtain a second label of the detection object;

[0165] The sixth processing submodule is used to process the detection image using the second label of the detection object to obtain a sub-image corresponding to the detection object.

[0166] In some embodiments, the second processing module 1002 includes:

[0167] A first detection submodule is used to detect the image to be detected from at least two detection dimensions to obtain a detection result of the object to be detected in each of the detection dimensions;

[0168] In some embodiments, the detection result includes location information and category information;

[0169] A seventh processing submodule, configured to associate the objects to be detected in the at least two detection dimensions based on the position information, and obtain an association result corresponding to the associated object;

[0170] In some embodiments, the association result includes category information corresponding to the associated object in each of the detection dimensions;

[0171] An eighth processing submodule, configured to verify the association result corresponding to the associated object based on the category association information, and obtain a target detection result corresponding to the object to be detected;

[0172] In some embodiments, the category association information is an association relationship between category information of at least two detection dimensions.

[0173] In some embodiments, the seventh processing submodule includes:

[0174] A first determining submodule, configured to determine the object to be detected in the at least two detection dimensions with position association as the associated object based on the position information of the object to be detected in the at least two detection dimensions;

[0175] The ninth processing submodule is used to establish a corresponding relationship between the category information of the associated objects in each of the detection dimensions to obtain the association result.

[0176] In some embodiments, the category association information includes an association relationship between each of the preset objects in a plurality of preset objects, and the association relationship includes category information corresponding to the preset object in each of the detection dimensions; the eighth processing submodule includes:

[0177] a tenth processing submodule, configured to compare the correspondence between the category information of the associated objects in each of the detection dimensions in each of the association results with the category information corresponding to the plurality of preset objects in each of the detection dimensions;

[0178] The second determination submodule is used to, when the comparison results are consistent, use the detection result of the object to be detected corresponding to the category information of at least two detection dimensions in the associated results as the target detection result corresponding to the object to be detected.

[0179] In some embodiments, the at least two detection dimensions include a defect detection dimension and a component detection dimension, and the first detection submodule includes:

[0180] A second detection submodule is used to detect the image to be detected from the defect detection dimension to obtain a defect detection result of the defect object in the defect detection dimension;

[0181] In some embodiments, the defect detection result includes location information of the defect object and a defect category of the defect object;

[0182] A third detection submodule is used to detect the image to be detected from the component detection dimension to obtain a component detection result of the component object in the component detection dimension;

[0183] In some embodiments, the component detection result includes location information of the component object and a component category of the component object;

[0184] Correspondingly, the seventh processing submodule includes:

[0185] A fifth processing submodule, configured to associate the defect object with the component object based on the position information of the defect object and the position information of the component object, and obtain an association result corresponding to the associated object;

[0186] In some embodiments, the association result includes an association relationship between a defect category of the defect object and a component category of the component object;

[0187] Correspondingly, the eighth processing submodule includes:

[0188] A sixth processing submodule, configured to verify the association relationship between the defect category of the defect object and the component category of the component object based on preset category association information, and obtain a target detection result corresponding to the object to be detected;

[0189] In some embodiments, the category association relationship is an association relationship between the defect category and the component category.

[0190] An embodiment of the present application provides an electronic device, Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Fig.11 As shown, the electronic device 1100 includes: a processor 1101, a memory 1102 and a communication bus 1103;

[0191] The communication bus 1103 is used to realize the communication connection between the processor 1101 and the memory 1102;

[0192] The processor 1101 is used to execute the program in the memory 1102 to implement the above-mentioned fall detection method.

[0193] An embodiment of the present application provides a computer-readable storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the above-mentioned fall detection method.

[0194] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned sequence numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0195] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0196] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0197] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0198] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0199] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0200] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0201] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image detection method, the method comprising: Processing the image to be processed based on the first segmentation model to obtain an image to be detected corresponding to the object to be detected, wherein the image to be processed is acquired when the object to be detected is located in the scene where the object to be detected is located; The first label used for training the first segmentation model is determined based on the second segmentation model, the second segmentation model is obtained by training multiple fused images carrying the second label, the fused image is formed by fusing a sub-image corresponding to the detection object and sub-images corresponding to other scenes, and the sub-image corresponding to the detection object is determined based on the second label; The image to be detected is detected to obtain a detection result corresponding to the object to be detected.

2. According to the method of claim 1, the first segmentation model is trained by multiple processed images carrying the first label, and the first label is obtained by predicting the processed image based on the second segmentation model.

3. According to the method of claim 1, the training method of the first segmentation model comprises: Performing model training based on the multiple fused images carrying the second label to obtain the second segmentation model; Using the second segmentation model to predict the multiple processed images, to obtain multiple processed images carrying prediction labels; For each of the processed images carrying the predicted label, obtaining an adjustment result for the predicted label to obtain a processed image carrying the first label; Model training is performed based on multiple processed images carrying the first label to obtain the first segmentation model.

4. The method according to claim 1, wherein the method for acquiring the fused image comprises: Acquire a detection image; wherein the detection image includes the detection object; Identify the detection image to determine the position information of the detection object in the detection image; Based on the position information of the detection object and the detection image, generating a sub-image corresponding to the second label and the detection object; The sub-image corresponding to the other scene and the sub-image corresponding to the detection object are synthesized into the fused image, and the fused image carries the second label.

5. The method according to claim 4, wherein generating the second label and the sub-image corresponding to the detection object based on the position information of the detection object and the detection image comprises: Based on the position information of the detection object, binarize the detection object to obtain a second label of the detection object; The detection image is processed using the second label of the detection object to obtain a sub-image corresponding to the detection object.

6. The method according to claim 1, wherein the detecting based on the image to be detected to obtain the detection result corresponding to the object to be detected comprises: Detect the image to be detected from at least two detection dimensions to obtain a detection result of the object to be detected in each detection dimension; the detection result includes position information and category information; Associating the objects to be detected in the at least two detection dimensions based on the position information to obtain an association result corresponding to the associated object; the association result includes category information corresponding to the associated object in each of the detection dimensions; The association result corresponding to the associated object is verified based on the category association information to obtain the target detection result corresponding to the object to be detected; the category association information is the association relationship between category information of at least two detection dimensions.

7. The method according to claim 6, wherein the step of associating the objects to be detected in the at least two detection dimensions based on the position information to obtain an association result corresponding to the associated objects comprises: Based on the position information of the object to be detected in the at least two detection dimensions, determining the object to be detected in the at least two detection dimensions with position association as the associated object; A corresponding relationship between the category information of the associated objects in each of the detection dimensions is established to obtain the association result.

8. The method according to claim 6, wherein the category association information includes an association relationship between each of the preset objects in a plurality of preset objects, the association relationship includes category information corresponding to the preset object in each of the detection dimensions, and the step of verifying the association result corresponding to the associated object based on the category association information to obtain the target detection result corresponding to the object to be detected includes: Comparing the corresponding relationship between the category information of the associated objects in each of the detection dimensions in each of the association results with the category information corresponding to the plurality of preset objects in each of the detection dimensions; When the comparison results are consistent, the detection result of the object to be detected corresponding to the category information of at least two detection dimensions in the associated results is used as the target detection result corresponding to the object to be detected.

9. The method according to any one of claims 6 to 8, wherein the at least two detection dimensions include a defect detection dimension and a component detection dimension; The detecting the image to be detected from at least two detection dimensions to obtain a detection result of the object to be detected in each detection dimension includes: Detect the image to be detected from the defect detection dimension to obtain a defect detection result of the defect object in the defect detection dimension; The defect detection result includes the position information of the defect object and the defect category of the defect object; the image to be detected is detected from the component detection dimension to obtain the component detection result of the component object in the component detection dimension; the component detection result includes the position information of the component object and the component category of the component object; The associating the objects to be detected in the at least two detection dimensions based on the position information to obtain an association result corresponding to the associated object includes: associating the defect object with the component object based on the position information of the defect object and the position information of the component object to obtain an association result corresponding to the associated object; the association result includes an association relationship between a defect category of the defect object and a component category of the component object; The verifying the association result corresponding to the associated object based on the preset category association information to obtain the target detection result corresponding to the object to be detected includes: verifying the association relationship between the defect category of the defect object and the component category of the component object based on the preset category association information to obtain the target detection result corresponding to the object to be detected; the category association relationship is the association relationship between the defect category and the component category.

10. An image detection device, comprising: A first processing module is used to process the image to be processed based on the first segmentation model to obtain an image to be detected corresponding to the object to be detected, wherein the image to be processed is acquired when the object to be detected is located in the scene where the object to be detected is located; The first label used for training the first segmentation model is determined based on a second segmentation model, the second segmentation model is obtained by training a plurality of fused images carrying masks, the fused image is formed by fusing a sub-image corresponding to the detection object and sub-images corresponding to other scenes, and the sub-image corresponding to the detection object is determined based on the mask; The second processing module is used to detect the image to be detected and obtain the detection result corresponding to the object to be detected.

Images