Object detection method, device, equipment and storage medium

Abstract

The embodiment of the present application provides an object detection method, device, equipment and storage medium, the object includes: obtaining a first image and a second image of the object to be detected, the first image is a two-dimensional image, and the second image at least includes a color normal vector image synthesized by photometric stereo; inputting the first image into a two-dimensional target detection model to output a first detection result, and inputting the second image into a photometric stereo target detection model to output a second detection result, the first detection result is used to indicate the position coordinates and confidence probability of each defect detected from the first image, and the second detection result is used to indicate the position coordinates and confidence probability of each defect detected from the second image; inputting the first detection result and the second detection result into a joint processing model to compare with a preset threshold to output the quality inspection result of the object to be detected. The technical solution provided by the embodiment of the present application is used to improve the accuracy of object detection.

Description

Technical Field

[0001] The present application relates to the field of industrial detection technology, and in particular to an object detection method, device, equipment and storage medium. Background Art

[0002] In industrial manufacturing scenarios, emerging technologies such as artificial intelligence and deep learning are often used to perform fully automated quality inspections on the appearance of products to ensure product consistency, yield, and safety. The following process is often used to detect appearance defects of objects: an image of the object is generated through a lens, and then the image is input into a trained target detection model or target segmentation model to output abnormal areas in the image, thereby achieving the goal of appearance quality inspection.

[0003] In real scenes, objects are generally three-dimensional, but when they are imaged through a lens, a two-dimensional image is generated, which is equivalent to a single-angle projection of the object's defects. This makes it difficult to identify some relatively subtle defects and deep defects, which can easily lead to missed inspections.

[0004] Therefore, there is an urgent need for a detection method that can improve the accuracy of object detection. Summary of the invention

[0005] The embodiments of the present application provide an object detection method, apparatus, device and storage medium for improving the accuracy of object detection.

[0006] In view of this, on the one hand, the present application provides an object detection method, including: acquiring a first image and a second image of an object to be detected, the first image being a two-dimensional image, and the second image at least including a color normal vector image synthesized by photometric stereo; inputting the first image into a two-dimensional target detection model to output a first detection result, and inputting the second image into a photometric stereo target detection model to output a second detection result, the first detection result being used to indicate the position coordinates and confidence probabilities of each defect detected from the first image, and the second detection result being used to indicate the position coordinates and confidence probabilities of each defect detected from the second image; inputting the first detection result and the second detection result into a joint processing model for comparison with a preset threshold to output a quality inspection result of the object to be detected, the preset threshold including at least one of a confidence probability threshold, a position coordinate quantity threshold, and a preset range of position coordinates.

[0007] On the other hand, the present application provides an object detection device, comprising: an acquisition module, for acquiring a first image and a second image of an object to be detected, the first image being a two-dimensional image, the second image at least comprising a color normal vector image synthesized by photometric stereo;

[0008] A processing module, used to input the first image into a two-dimensional object detection model to output a first detection result, and input the second image into a photometric stereo object detection model to output a second detection result, wherein the first detection result is used to indicate the position coordinates and confidence probability of each defect detected from the first image, and the second detection result is used to indicate the position coordinates and confidence probability of each defect detected from the second image;

[0009] The output module is used to input the first detection result and the second detection result into a joint processing model for comparison with a preset threshold to output the quality inspection result of the object to be detected. The preset threshold includes at least one of a confidence probability threshold, a position coordinate quantity threshold and a preset range of position coordinates.

[0010] In one possible design, in another implementation of another aspect of the embodiment of the present application, the object detection device also includes an estimation module, which is used to calculate, based on the position coordinates indicated by the second detection result and the color normal vector image, a relative depth set of each pixel point within the position coordinates indicated by the second detection result using the path integral principle; and select the maximum relative depth in the relative depth set as the estimated depth of the defect indicated by the second detection result.

[0011] In one possible design, in another implementation of another aspect of the embodiment of the present application, the output module is specifically used to compare the quality inspection result of the object to be inspected by jointly processing the model output of the first detection result, the second detection result and the estimated depth input with the preset threshold, and the preset threshold also includes a threshold for the estimated depth.

[0012] In one possible design, in another implementation of another aspect of the embodiment of the present application, the input module is specifically configured to output the quality inspection result of the target work as quality inspection failure when the confidence probability indicated by the first detection result is greater than the first threshold or the confidence probability indicated by the second detection result is greater than the second threshold and the estimated depth is greater than the third threshold;

[0013] When the confidence probability indicated by the first detection result is less than or equal to the first threshold, the confidence probability indicated by the second detection result is less than or equal to the second threshold, and the estimated depth is less than or equal to the third threshold, the quality inspection result of the target work is output as quality inspection passed.

[0014] In one possible design, in another implementation of another aspect of the embodiment of the present application, the acquisition module is specifically used to acquire an image set of the object to be detected under different lighting conditions through a target lens and at least three light sources, and the at least three light sources are light sources located at different positions relative to the object to be detected;

[0015] Any image in the image set is selected as the first image, and images generated when the at least three light sources illuminate the object to be detected individually are selected to synthesize the second image through photometric stereo.

[0016] In one possible design, in another implementation of another aspect of the embodiment of the present application, the acquisition module is specifically used to synthesize the second image through photometric stereo based on the images generated when the at least three light sources separately illuminate the object to be detected, and the reflection directions of the at least three light sources, wherein the incident direction is determined by light source calibration.

[0017] In one possible design, in another implementation of another aspect of the embodiment of the present application, when the number of the light sources is 4, the acquisition module is specifically configured to generate a third image of the object to be detected through the target lens when the first light source illuminates the object to be detected;

[0018] When the second light source illuminates the object to be detected, a fourth image of the object to be detected is generated through the target lens;

[0019] When the third light source illuminates the object to be detected, a fifth image of the object to be detected is generated through the target lens;

[0020] When the fourth light source illuminates the object to be detected, a sixth image of the object to be detected is generated through the target lens;

[0021] When the first light source, the second light source, the third light source and the fourth light source all illuminate the object to be detected, a seventh image of the object to be detected is generated through the target lens; wherein the image set includes the third image, the fourth image, the fifth image, the sixth image and the seventh image;

[0022] The first light source, the second light source, the third light source and the fourth light source are light sources located at different positions relative to the object to be detected;

[0023] The acquisition module is specifically configured to select the seventh image as the first image;

[0024] The third image, the fourth image, the fifth image and the sixth image are synthesized into the second image through photometric stereo.

[0025] In one possible design, in another implementation of another aspect of the embodiment of the present application, the target lens is a telecentric lens or a telephoto lens;

[0026] The at least three light sources are telecentric illumination light sources with uniform intensity or long-distance point light sources.

[0027] In one possible design, in another implementation of another aspect of the embodiment of the present application, the second image also includes a reflectance grayscale image synthesized by photometric stereo.

[0028] In one possible design, in another implementation of another aspect of the embodiment of the present application, the acquisition module is also used to acquire a first training sample set and a second training sample set, an initial two-dimensional target detection model and an initial photometric stereo target detection model, wherein the first training sample set marks surface defects and the second training sample set marks depth defects; the object detection device also includes a training module, which is used to use the first training sample set to perform supervised training on the initial two-dimensional target detection model to obtain the two-dimensional target detection model, and use the second training sample set to perform supervised training on the initial photometric stereo target detection model to obtain the photometric stereo target detection model.

[0029] In one possible design, in another implementation of another aspect of the embodiment of the present application, the initial two-dimensional target detection model is a cascaded convolutional neural network for region detection (Cascade Region-based Convolutional Neural Networks, Cascade RCNN), a fast convolutional neural network for region detection (Faster Region-based Convolutional Neural Networks, Faster RCNN) or a self-attention model for target detection (DEtection Transformer, DETR);

[0030] The initial photometric stereo object detection model is Cascade RCNN, Faster RCNN or DETR.

[0031] In one possible design, in another implementation of another aspect of an embodiment of the present application, the feature extraction backbone network in the initial two-dimensional target detection model and the initial photometric stereo target detection model is a high-resolution network (High-Resolution netV2P, HRNetV2P) with a feature pyramid, a residual neural network (ResidualNeural Network, ResNet) or a hierarchical visual self-attention model (SwinTransformer) based on a moving window.

[0032] Another aspect of the present application provides a computer device, including: a memory, a processor, and a bus system;

[0033] Wherein, the memory is used to store programs;

[0034] The processor is used to execute the program in the memory, and the processor is used to execute the above-mentioned methods according to the instructions in the program code;

[0035] The bus system is used to connect the memory and the processor so that the memory and the processor can communicate with each other.

[0036] Another aspect of the present application provides a computer-readable storage medium, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer is enabled to execute the above-mentioned methods.

[0037] Another aspect of the present application provides a computer program product or a computer program, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided by the above aspects.

[0038] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages: combining two-dimensional target detection with photometric stereo detection, so that both surface defects of objects and deep defects of scratches on objects can be detected, thereby improving the accuracy of object detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A schematic diagram of the architecture of the application system in the embodiment of the present application;

[0040] Figure 2 A schematic diagram of a network architecture of Cascade RCNN in an embodiment of the present application;

[0041] Figure 3 A schematic diagram of a network architecture of HRNetV2P in an embodiment of the present application;

[0042] Figure 4 A schematic diagram of a flow chart of an object detection method in an embodiment of the present application;

[0043] Figure 5 A schematic diagram of the structure of an imaging machine in an embodiment of the present application;

[0044] Figure 6 A schematic diagram of an embodiment of an object detection method in an embodiment of the present application;

[0045] Figure 7 An imaging schematic diagram of light source calibration in an embodiment of the present application;

[0046] Figure 8 A schematic diagram of the light source direction and the light source reflection direction during the light source calibration process in the embodiment of the present application;

[0047] Fig. 9 It is a schematic diagram of the light source direction, reflection direction and normal vector in the embodiment of the present application;

[0048] Fig.10 A schematic diagram of photometric stereo image synthesis in an embodiment of the present application;

[0049] Fig.11 This is another schematic diagram of an embodiment of the object detection method in the embodiment of the present application;

[0050] Fig.12 A schematic diagram of a height field representation in an embodiment of the present application;

[0051] Fig.13 A schematic diagram of path integration in an embodiment of the present application;

[0052] Fig.14 A schematic diagram of solving the relative height by path integral in an embodiment of the present application;

[0053] Fig.15 A schematic diagram of an embodiment of an object detection device in an embodiment of the present application;

[0054] Fig.16 It is a schematic diagram of another embodiment of the object detection device in the embodiment of the present application;

[0055] Fig.17 It is a schematic diagram of another embodiment of the object detection device in the embodiment of the present application;

[0056] Fig.18 It is a schematic diagram of another embodiment of the object detection device in the embodiment of the present application;

[0057] Fig.19 FIG. 2 is a schematic diagram of another embodiment of the object detection device in the embodiment of the present application. DETAILED DESCRIPTION

[0058] The embodiments of the present application provide an object detection method, apparatus, device and storage medium for improving the accuracy of object detection.

[0059] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "corresponding to" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0060] For ease of understanding, some professional terms involved in this application are explained below.

[0061] Defect Detection: Defect is also called Anomaly Detection. Its main task is to determine whether the image is defective.

[0062] Object Detection: An important application of computer vision, it is used to detect frames containing specific targets from images, such as faces, vehicles, or buildings, depending on the actual application scenario. That is, object detection focuses on specific object targets and requires obtaining the category information and location information (classification and localization) of this target at the same time. The object detection model needs to have a full understanding of the foreground and background of the image in order to separate the target of interest from the background and determine the description of this target (category and location). The output of the object detection model is a list, and each item in the list uses an array to give the category and location of the detected target (commonly represented by the coordinates of a rectangular detection box).

[0063] Machine Learning (ML): The core of artificial intelligence is the fundamental way to make computers intelligent. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. In the embodiment of the present application, the target detection model can be a machine learning model.

[0064] Loss function: It is used to measure the inconsistency between the predicted value of the model and the true value (labeled value). It is usually associated with the optimization problem as a learning criterion, that is, model training is achieved by minimizing the loss function.

[0065] Convolutional Neural Networks: Convolutional Neural Networks (CNN) is a type of feedforward neural network with deep structure and convolutional calculations, and is one of the representative algorithms of deep learning. RCNN is a convolutional neural network based auditory region detection network, which specifically includes two types: FasterRCNN and Cascade RCNN.

[0066] Back propagation: Forward propagation refers to the feedforward processing of the model, while back propagation is the opposite of forward propagation, which refers to updating the weight parameters of each layer of the model according to the output of the model. For example, if the model includes input layer, hidden layer and output layer, then forward propagation refers to processing in the order of input layer-hidden layer-output layer, and back propagation refers to updating the weight parameters of each layer in the order of output layer-hidden layer-input layer.

[0067] Confidence: Also known as confidence probability, it is used to indicate the probability that the corresponding result is credible. For example, if the confidence of a prediction box that includes a target is 80%, it means that the probability that the prediction box actually includes the target is 80%.

[0068] Photometric Stereo (PS): Photometric stereo is a method for estimating surface geometry using multiple light source directions. The purpose of this method is to reconstruct the normal vector of the object surface and the reflectivity of different surface points of the object. It does not need to consider the image matching problem like traditional geometric reconstruction methods (such as stereo matching), because all you need to do is collect more than three images of the object illuminated by light from different directions. In this process, neither the object nor the camera moves, so the images are naturally aligned, which makes the whole process very simple.

[0069] RGB: It stands for the colors of the three channels of red, green and blue. It refers to the changes of the three color channels of red (R), green (G) and blue (B) and their superposition to obtain various colors. It is the storage format and display format of photos after camera imaging. In this article, it also refers to general photos such as two-dimensional images that are distinguished from photometric stereo normal vector images.

[0070] In industrial manufacturing scenarios, emerging technologies such as artificial intelligence and deep learning are often used to perform fully automated quality inspections on the appearance of products to ensure product consistency, yield, and safety. The following process is often used for the detection of appearance defects of objects: an image of the object is generated through a lens, and then the image is input into a trained target detection model or target segmentation model to output abnormal areas in the image, thereby achieving the goal of appearance quality inspection. In actual scenarios, objects are generally three-dimensional, and a two-dimensional image is generated when imaging through a lens, which is equivalent to a single-angle projection of the object's defects. In this way, some subtle defects and deep defects will not be easy to distinguish, which may easily lead to missed detection. Therefore, there is an urgent need for a detection method that can improve the accuracy of object detection. In order to solve this problem, the present application provides the following technical solutions: obtain a first image and a second image of the object to be detected, the first image is a two-dimensional image, and the second image at least includes a color normal vector image synthesized by photometric stereo; input the first image into a two-dimensional target detection model to output a first detection result, and input the second image into a photometric stereo target detection model to output a second detection result, the first detection result is used to indicate the position coordinates and confidence probability of each defect detected from the first image, and the second detection result is used to indicate the position coordinates and confidence probability of each defect detected from the second image; input the first detection result and the second detection result into a joint processing model for comparison with a preset threshold to output the quality inspection result of the object to be detected, the preset threshold includes at least one of a confidence probability threshold, a position coordinate quantity threshold, and a preset range of position coordinates. In this way, two-dimensional target detection is combined with photometric stereo detection, so that both surface defects of objects and deep defects of scratches on objects can be detected, thereby improving the accuracy of object detection.

[0071] The object detection method, apparatus, device and storage medium provided in the embodiments of the present application can improve the accuracy of object detection. The following describes an exemplary application of the electronic device provided in the embodiments of the present application. The electronic device provided in the embodiments of the present application can be implemented as various types of user terminals or as a server.

[0072] By running the object detection solution provided in the embodiment of the present application, the electronic device can improve the accuracy of object detection, that is, improve the object detection performance of the electronic device itself, and is suitable for multiple application scenarios of object detection. For example, steel plate defect detection, floor defect detection, printed circuit board (PCB) defect detection, film defect detection, lamp bead defect detection, metal bar end face detection, fabric wrinkle grade assessment, surface defect automatic quality inspection instrument for industrial parts, etc.

[0073] See also Figure 1 , Figure 1It is an optional architecture diagram under an application scenario of the object detection solution provided in the embodiment of the present application. To realize supporting an object detection application, the terminal device 100 (exemplarily showing the terminal device 1001 and the terminal device 1002) is connected to the server 300 through the network 200, and the server 300 is connected to the database 400. The network 200 can be a wide area network or a local area network, or a combination of the two. The client for implementing the object detection solution is deployed on the terminal device 100, wherein the client can be run on the terminal device 100 in the form of a browser, or can be run on the terminal device 100 in the form of an independent application (application, APP), etc. The specific presentation form of the client is not limited here. The server 300 involved in this application can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms. The terminal device 100 can be a smart phone, a tablet computer, a laptop, a PDA, a personal computer, a smart TV, a smart watch, a vehicle-mounted device, a wearable device, etc., but is not limited thereto. The terminal device 100 and the server 300 can be directly or indirectly connected through the network 200 by wired or wireless communication, and this application does not limit this. The number of servers 300 and terminal devices 100 is also not limited. The solution provided in this application can be completed independently by the terminal device 100, or by the server 300, or by the terminal device 100 and the server 300 in cooperation, and this application does not make specific limitations. Among them, the database 400, in short, can be regarded as an electronic filing cabinet-a place for storing electronic files, and users can add, query, update, delete, and other operations on the data in the file. The so-called "database" is a collection of data that is stored together in a certain way, can be shared with multiple users, has as little redundancy as possible, and is independent of the application program. A database management system (DBMS) is a computer software system designed for managing databases, generally with basic functions such as storage, interception, security, and backup.Database management systems can be classified according to the database models they support, such as relational, Extensible Markup Language (XML); or according to the types of computers they support, such as server clusters, mobile phones; or according to the query language used, such as Structured Query Language (SQL), XQuery; or according to performance impulse focus, such as maximum scale, maximum operating speed; or other classification methods. Regardless of the classification method used, some DBMSs can cross categories, for example, supporting multiple query languages ​​at the same time. In the present application, database 400 can be used to store training sample sets and images generated after imaging of objects. Of course, the storage location of the training sample sets is not limited to the database, for example, it can also be stored in the terminal device 100, the blockchain or the distributed file system of the server 300.

[0074] In some embodiments, the server 300 may execute the object detection method provided in the embodiment of the present application and the training method of the two-dimensional target detection model and the photometric stereo target detection model in the object detection. When executing the training method of the two-dimensional target detection model and the photometric stereo target detection model, the specific process may be as follows: obtain the first training sample set and the second training sample set corresponding to the labeled frame from the terminal device 100 and / or the database 400, detect and process the first training sample set through the two-dimensional target detection model to obtain the prediction frame including the object defect in the first training sample set, and determine the loss value corresponding to the prediction frame according to the loss function including the pre-designed loss factors (such as the interval value and the distance), and then train the two-dimensional target detection model; at the same time, detect and process the second training sample set through the photometric stereo target detection model to obtain the prediction frame including the object defect in the second training sample set, and determine the loss value corresponding to the prediction frame according to the loss function including the pre-designed loss factors (such as the interval value and the distance), and then train the photometric stereo target detection model. In this embodiment, the first training sample set at least annotates the surface defects, and the second training sample set at least annotates the depth defects. At the same time, the first training sample set and the second training sample set can be the same batch of images or different images. That is, the two-dimensional object detection model and the photometric stereo object detection model are trained independently, and do not affect the adjustment of parameters of each other. In this way, the generalization ability of the two-dimensional object detection model and the photometric stereo object detection model can be improved.

[0075] When the server 300 trains the two-dimensional target detection model and the photometric stereo target detection model, the initial model architecture of the two-dimensional target detection model can be Cascade RCNN, Faster RCNN or DETR, and the initial model architecture of the photometric stereo target detection model can be Cascade RCNN, Faster RCNN or DETR. That is, the two-dimensional target detection model and the photometric stereo target detection model are trained in parallel, and the two-dimensional target detection model and the photometric stereo target detection model can select different model architectures according to their specific requirements, so as to realize the flexible application of the target detection model in the object detection. In an exemplary scheme, the initial model architecture of the two-dimensional target detection model can be Cascade RCNN, and the initial model architecture of the photometric stereo target detection model can be Faster RCNN. No specific limitation is made here. The two-dimensional target detection model and the photometric stereo target detection model both have a feature extraction backbone network under different model architectures. In this case, the feature extraction backbone network can be a high-resolution network with a feature pyramid (High-Resolution netV2P, HRNetV2P), a residual neural network (Residual Neural Network, ResNet) or a hierarchical visual self-attention model based on a moving window (Swin Transformer). The following takes Cascade RCNN and the feature extraction backbone network HRNetV2P as an example to illustrate the two-dimensional target detection model and the photometric stereo target detection model. Figure 2 As shown, the Cascade RCNN includes an input interface, and then the image is input into the Cascade RCNN through the input interface, and then the feature representation map of the image is obtained through the feature extraction backbone network (conv) of the Cascade RCNN; then the feature representation map is input into the local area feature extractor (pool) to extract the local feature representation map in the image; then the local feature representation map is input into the network head for feature transformation to enrich the feature representation of the image; finally, the classification result of the image can be obtained through the classifier and the preset bounding box. In this embodiment, the specific network structure of the HRNetV2P can be as follows Figure 3As shown in the figure, a multi-resolution block consists of a multi-resolution group convolution and a multi-resolution convolution. Multi-resolution group convolution is a simple extension of group convolution. It divides the input channels into several groups and applies regular convolutions at different spatial scales and between each group. Multi-resolution convolution connects different branches in a fully connected manner like regular convolution. At the same time, there are different resolutions between the channels of multi-resolution convolution. The connection between the input and output channels needs to reduce or increase the resolution due to different resolutions. The resolution reduction is achieved through one or more convolutions with a stride of 2 and a kernel size of 3x3, and the resolution increase is achieved through upsampling with bilinear interpolation; then the obtained high-resolution feature map is averaged and pooled to obtain four scales of expression; when performing feature fusion, multi-scale feature fusion is achieved and directly concatenated (concat).

[0076] After the two-dimensional object detection model and the photometric stereo object detection model are trained, the server 300 can save the two-dimensional object detection model and the photometric stereo object detection model locally, thereby providing the terminal device 100 with a remote object detection function. For example, the server 300 can receive the first image and the second image of the object to be detected sent by the terminal device 100, and detect the first image through the two-dimensional object detection model to obtain the prediction box (i.e., position coordinates) and the corresponding confidence probability corresponding to the defect in the first image; at the same time, the second image is detected through the photometric stereo object detection model to obtain the prediction box (i.e., position coordinates) and the corresponding confidence probability corresponding to the defect in the second image. Then, the server 300 determines the quality inspection result of the object to be detected by jointly processing the prediction box (i.e., position coordinates) and the corresponding confidence probability corresponding to the defect in the first image and the prediction box (i.e., position coordinates) and the corresponding confidence probability corresponding to the defect in the second image, and finally sends the quality inspection result to the terminal device 100, so that the terminal device 100 displays the quality inspection result in the graphical interface 110 (graphical interface 1101 and graphical interface 1102 are shown as examples). It can be understood that the server 300 can also send the prediction box (i.e., position coordinates) and the corresponding confidence probability corresponding to the defect in the first image and the prediction box (i.e., position coordinates) and the corresponding confidence probability corresponding to the defect in the second image to the terminal device 100, so that the terminal device 100 displays the prediction box (i.e., position coordinates) and the corresponding confidence probability corresponding to the defect in the first image and the prediction box (i.e., position coordinates) and the corresponding confidence probability corresponding to the defect in the second image in the graphical interface 110 (graphical interface 1101 and graphical interface 1102 are shown as examples).

[0077] The server 300 can also send (deploy) the trained two-dimensional object detection model and photometric stereo object detection model to the terminal device 100, so as to realize object detection locally on the terminal device 100. For example, the terminal device 100 can obtain the first image and the second image of the object to be detected in real time or obtain the first image and the second image of the object to be detected from other devices, and detect and process the first image through the two-dimensional object detection model to obtain the prediction box (i.e., position coordinates) and the corresponding confidence probability corresponding to the defect in the first image; at the same time, the second image is detected and processed through the photometric stereo object detection model to obtain the prediction box (i.e., position coordinates) and the corresponding confidence probability corresponding to the defect in the second image. Then, the terminal device 100 determines the quality inspection result of the object to be detected by jointly processing the prediction box (i.e., position coordinates) and the corresponding confidence probability corresponding to the defect in the first image and the prediction box (i.e., position coordinates) and the corresponding confidence probability corresponding to the defect in the second image; finally, the terminal device 100 displays the quality inspection result in the graphical interface 110 (graphical interface 1101 and graphical interface 1102 are shown as examples). It can be understood that the terminal device 100 can also display the prediction box (i.e., position coordinates) and the corresponding confidence probability corresponding to the defect in the first image and the prediction box (i.e., position coordinates) and the corresponding confidence probability corresponding to the defect in the second image in the graphical interface 110 (graphical interface 1101 and graphical interface 1102 are shown as examples).

[0078] Based on the above system, please refer to Figure 4 As shown, an execution process of the object detection method in this application can be as follows:

[0079] Step 1: image the object to be detected to generate an image set, and generate a first image and a second image based on the image set. Specifically, the object to be detected is placed on a machine that has been calibrated by a light source to generate multiple images through a target lens. In an exemplary solution, if the machine is Figure 5 As shown, it includes four light sources in different directions, a target lens, and a corresponding observation platform, and five different images of the object to be detected are generated by overhead shooting, wherein the four images are respectively the image taken when the front light source is illuminated alone, the image taken when the rear light source is illuminated alone, the image taken when the left light source is illuminated alone, the image taken when the right light source is illuminated alone, and the image taken when the four light sources are illuminated simultaneously. Among them, the image taken when the four light sources are illuminated simultaneously can be selected as the first image, and then the remaining images are synthesized by photometric stereo to generate the second image. It can be understood that the second image at least includes a color normal vector image. At the same time, the second image can also include a reflectivity grayscale image.

[0080] Step 2: Detect the first image through the two-dimensional target detection model to obtain a first detection result; detect the second image through the photometric target detection model to obtain a second detection result.

[0081] Step 3: Calculate the second detection result and the second image to obtain an estimated depth.

[0082] Step 4: The first detection result, the second detection result and the estimated depth are determined by a joint processing model to obtain a quality inspection result of the object to be detected.

[0083] It can be understood that in the specific implementation of the present application, related data such as the first image and the second image are involved. When the above embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0084] In combination with the above introduction, the object detection method in this application will be introduced by taking the terminal device executing the object detection method as an example. Figure 6 , an embodiment of the object detection method in the embodiment of the present application includes:

[0085] 601. Acquire a first image and a second image of an object to be detected, where the first image is a two-dimensional image, and the second image at least includes a color normal vector image synthesized by photometric stereo.

[0086] The terminal device obtains a first image and a second image of the object to be detected. Specifically, the terminal device generates a plurality of image sets through a target lens, then selects an image from the image set as the first image, and selects at least three light sources from the image set to perform photometric stereo synthesis of the second image when the images are generated when the object to be detected is illuminated separately.

[0087] In an exemplary embodiment, Figure 5 As shown, it includes four light sources in different directions, a target lens, and a corresponding observation platform, and five different images of the object to be detected are generated by overhead shooting, wherein the four images are respectively the image taken when the front light source is illuminated alone, the image taken when the rear light source is illuminated alone, the image taken when the left light source is illuminated alone, the image taken when the right light source is illuminated alone, and the image taken when the four light sources are illuminated simultaneously. Among them, the image taken when the four light sources are illuminated simultaneously can be selected as the first image, and then the remaining images are synthesized by photometric stereo to generate the second image.

[0088] In this embodiment, the Figure 5Before the imaging machine shown in the figure obtains the image set of the object to be detected, the light source of the imaging machine needs to be calibrated. That is, the purpose of light source calibration is to estimate the incident direction of different parallel light sources, which is the first step of the photometric stereo algorithm. When calibrating the light source in the imaging machine, one method of calibrating the light source is to use a metal ball, such as Figure 7 As shown in the figure, the brightest point on the photo of the metal ball indicates the direction of the light source (due to the shape of the light source, it is not necessarily a point light source. Generally, the center of the brightest area can be taken as an approximation of the light source direction). Figure 8 As shown in the reflection diagram, the normal vector under the current light source can be calculated based on the pixel coordinates of the brightest point in the image and the pixel coordinates of the metal ball in the image. Assume that the current reflection direction of the light source is R (it can be understood that R is a fixed direction) (P x ,P y ) is the pixel coordinate position of the brightest point in the image, (C x ,C y ) is the center point of the metal ball in the image, from which we can find the normal vector N = [N x ,N y .N z ] T .

[0089] Among them, N x =P x -C x ; N y =P y -C y ; N z γ in the equation is the radius of the metal ball. Since the modulus of the light source direction L and the reflection direction R of the light source are equal, that is, ||L||=||R||. At this time, there must be a light source L, R, such that R+L=N=2S; L=2S-R, where S is the projection of R on N, |N|=1, so we can get: Substituting the expression for L, we get: L = 2(R·N)NR. Since we only need to solve for the direction of L, that is, divide both sides of the equation by the modulus of L and R, the equation still holds. For the convenience of calculation, we assign R to get: At this time, by substituting the normal vector N into the above formula, the direction of the light source can be solved.

[0090] After the direction of the light source is calibrated, the process of the terminal device acquiring the photometric stereo composite image of the object to be detected can be as follows:

[0091] The schematic diagram of the light source direction, reflection direction and normal vector is as follows Fig. 9As shown, based on the schematic diagram and the surface reflectivity ρ of the object to be detected, the representation formula of the first pixel on the image of the object to be detected can be obtained: I(x,y)=ρ(x,y)L i L·N(x,y); in this formula, L i represents the intensity of the light source, L and N represent the direction vector (unit vector, the direction modulus is 1), and ρ is a constant. In this case, ρ and N can be combined into G to represent it. For an image, assuming that the light source intensity is 1, the image can be represented as I = L T G. Based on Figure 5 The scenario shown, for I = L T G is expressed by the least squares method to obtain the formula min G ||IL T G|| 2 , and then we get the formula ||IL T G|| 2 =I T I+G T LL T G-2G T LI. To find the minimum value of the expanded form of the above equation, we can find the partial differential with respect to G and set it to 0, which gives the formula 2LL T G-2LI=0; then solve the formula 2LL T G-2LI=0 can be obtained (the expression of G obtained by least squares): G=(LL T ) -1 LI.

[0092] Since the modulus of G is the reflectivity ρ, that is, ρ = ||G||, the normalized unit vector matrix based on this is the normal vector N, that is, Based on the above process, an exemplary solution is as follows: Fig.10 As shown in Figure 1, the input of photometric stereo image synthesis is grayscale images illuminated by four different light sources, and the output is a color normal vector map and a reflectivity grayscale map.

[0093] It can be understood that after the color normal vector image and the reflectance grayscale image are obtained by photometric stereo synthesis of the object to be detected, only the color normal vector image can be used as the second image, or both the color normal vector image and the reflectance grayscale image can be used as the second image.

[0094] In this embodiment, in order to ensure that the photometric stereo composite image can be better used to represent the surface information of the object to be detected, the target lens is a telecentric lens or a long-focus lens; the at least three light sources are telecentric illumination light sources with uniform intensity or long-distance point light sources.

[0095] 602. Input the first image into a two-dimensional target detection model to output a first detection result, and input the second image into a photometric stereo target detection model to output a second detection result, wherein the first detection result is used to indicate the position coordinates and confidence probability of each defect detected from the first image, and the second detection result is used to indicate the position coordinates and confidence probability of each defect detected from the second image.

[0096] After acquiring the first image and the second image, the terminal device respectively inputs the first image into the two-dimensional target detection model to output the first detection result, at which time the first detection result is used to indicate the position coordinates and corresponding confidence probability of the surface defect of the object to be detected; at the same time, the second image is input into the photometric stereo target detection model to output the second detection result, at which time the second detection result is used to indicate the position coordinates and corresponding confidence probability of the depth defect of the object to be detected. It can be understood that the position of the target lens and the position of the object to be detected have not changed, so the position coordinates corresponding to the first image and the position coordinates corresponding to the second image are the same in the same coordinate system. In an exemplary scheme, the first detection result may include the following information: position coordinate 1 (a1, b1, c1, d1), the corresponding confidence probability is 0.5; position coordinate 2 (a2, b2, c2, d2), the corresponding confidence probability is 0.3; position coordinate 3 (a3, b3, c3, d3), the corresponding confidence probability is 0.8. That is, the terminal device detects that there are 3 surface defects of the object to be detected. The second detection result may include the following information: position coordinate 4 (a4, b4, c4, d4), the corresponding confidence probability is 0.3; position coordinate 5 (a5, b5, c5, d5), the corresponding confidence probability is 0.6; position coordinate 6 (a6, b6, c6, d6), the corresponding confidence probability is 0.8. That is, the terminal device detects that there are 3 depth defects in the object to be detected.

[0097] In this embodiment, the input channel of the photometric stereo target detection model can be flexibly changed. For example, when the photometric stereo target detection model only includes the color normal vector image, its input channel is 3; when the photometric stereo target detection model includes the color normal vector image and the reflectivity grayscale image, its input channel is 4; when the photometric stereo target detection model includes the color normal vector image, the reflectivity grayscale image and the first image, its input channel is 7.

[0098] 603. Input the first detection result and the second detection result into a joint processing model for comparison with a preset threshold to output a quality inspection result of the object to be detected. The preset threshold includes at least one of a confidence probability threshold, a position coordinate quantity threshold, and a preset range of position coordinates.

[0099] The terminal device may input the first detection result and the second detection result into the joint processing model for comprehensive calculation to obtain the quality inspection result of the object to be detected. Specifically, the terminal device may compare the position coordinates of each defect in the first detection result with its corresponding confidence probability and a preset threshold, and at the same time compare the position coordinates of each defect in the second detection result and the corresponding confidence probability with the preset threshold.

[0100] In this embodiment, the preset threshold includes a threshold for the confidence probability of the first detection result, that is, to determine whether the possibility of the existence of defects in the object to be detected exceeds the threshold; the preset threshold also includes a threshold for the confidence probability of the second detection result, that is, to determine whether the possibility of the existence of defects in the object to be detected exceeds the threshold; the preset threshold also includes a threshold for the number of position coordinates in the first detection result and the second detection result, that is, to determine whether the defects that may exist in the object to be detected exceed the threshold; the preset threshold also includes a threshold for the position coordinates of each defect in the first detection result and the second detection result, that is, to determine whether the position where the defect may exist in the object to be detected is in an important part. It can be understood that the preset threshold includes at least one of the threshold for the confidence probability, the threshold for the number of position coordinates, and the preset range of the position coordinates.

[0101] In an exemplary solution, the preset threshold includes a threshold of the confidence probability, and the terminal device determines the quality inspection result of the object to be detected according to the confidence probability of each position coordinate in the first detection result and the confidence probability of each position coordinate in the second detection result. That is, when the confidence probability indicated by the first detection result is greater than the first threshold or the confidence probability indicated by the second detection result is greater than the second threshold, the quality inspection result of the target work is output as quality inspection failure; when the confidence probability indicated by the first detection result is less than or equal to the first threshold and the confidence probability indicated by the second detection result is less than or equal to the second threshold, the quality inspection result of the target work is output as quality inspection pass. It can be understood that the first detection result and the second detection result include multiple position coordinates (in this embodiment, the position coordinates are used to indicate a defective target to be determined as a defect), and when corresponding to multiple confidence probabilities, as long as there is a position coordinate with a confidence probability greater than the preset threshold, the quality inspection result of the object to be detected can be determined as not passed. For example, the first detection result may include the following information: position coordinate 1 (a1, b1, c1, d1), the corresponding confidence probability is 0.5; position coordinate 2 (a2, b2, c2, d2), the corresponding confidence probability is 0.3; position coordinate 3 (a3, b3, c3, d3), the corresponding confidence probability is 0.8. That is, the terminal device detects that there are 3 surface defect targets of the object to be detected. The second detection result may include the following information: position coordinate 4 (a4, b4, c4, d4), the corresponding confidence probability is 0.3; position coordinate 5 (a5, b5, c5, d5), the corresponding confidence probability is 0.6; position coordinate 6 (a6, b6, c6, d6), the corresponding confidence probability is 0.8. That is, the terminal device detects that there are 3 deep defect targets of the object to be detected. The confidence probability threshold for the surface defect is 0.5, and the confidence probability threshold for the depth defect is 0.3, so the terminal device confirms that the quality inspection result of the object to be inspected is that the quality inspection fails.

[0102] In another exemplary embodiment, the preset threshold includes a threshold of the number of position coordinates, and the terminal device determines the quality inspection result of the object to be inspected according to the number of each position coordinate in the first detection result and the number of each position coordinate in the second detection result. That is, when the sum of the number of each position coordinate indicated by the first detection result and the number of each position coordinate indicated by the second detection result is greater than the fourth threshold, the quality inspection result of the target work is output as quality inspection failure; when the sum of the number of each position coordinate indicated by the first detection result and the number of each position coordinate indicated by the second detection result is less than or equal to the fourth threshold, the quality inspection result of the target work is output as quality inspection passed. For example, the first detection result may include the following information: position coordinate 1 (a1, b1, c1, d1), the corresponding confidence probability is 0.5; position coordinate 2 (a2, b2, c2, d2), the corresponding confidence probability is 0.3; position coordinate 3 (a3, b3, c3, d3), the corresponding confidence probability is 0.8. That is, the terminal device detects that there are 3 surface defect targets of the object to be inspected. The second detection result may include the following information: position coordinate 4 (a4, b4, c4, d4), the corresponding confidence probability is 0.3; position coordinate 5 (a5, b5, c5, d5), the corresponding confidence probability is 0.6; position coordinate 6 (a6, b6, c6, d6), the corresponding confidence probability is 0.8. That is, the terminal device detects that there are 3 deep defect targets of the object to be detected, and the total number of defect targets detected by the terminal device for the object to be detected is 6. The threshold value for the number of position coordinates is set to 7, and the terminal device confirms that the quality inspection result of the object to be detected is passed.

[0103] In another exemplary embodiment, the preset threshold includes a threshold of the position coordinates, and the terminal device determines the quality inspection result of the object to be detected according to each position coordinate in the first detection result and each position coordinate in the second detection result. That is, when the first detection result indicates that each position coordinate and the second detection result indicate that each position coordinate are located in an important area (i.e., a preset range) of the object to be detected, the quality inspection result of the target work is output as quality inspection failure; when the first detection result indicates that each position coordinate and the second detection result indicate that each position coordinate are not in an important area (i.e., a preset range) of the object to be detected, the quality inspection result of the target work is output as quality inspection passed. For example, the first detection result may include the following information: position coordinate 1 (a1, b1, c1, d1), the corresponding confidence probability is 0.5; position coordinate 2 (a2, b2, c2, d2), the corresponding confidence probability is 0.3; position coordinate 3 (a3, b3, c3, d3), the corresponding confidence probability is 0.8. The second detection result may include the following information: position coordinate 4 (a4, b4, c4, d4), the corresponding confidence probability is 0.3; position coordinate 5 (a5, b5, c5, d5), the corresponding confidence probability is 0.6; position coordinate 6 (a6, b6, c6, d6), the corresponding confidence probability is 0.8. The important area of ​​the object to be detected is (A, B, C, D), where the position coordinate 1 is located in the important area, and the position coordinate 4 is located in the important area, then the terminal device confirms that the quality inspection result of the object to be detected is that the quality inspection fails.

[0104] It is understandable that the terminal device can implement any combination of the above three comparison schemes when comparing the first detection result and the second detection result with the preset threshold, as long as accurate detection of the object to be detected can be achieved, and the specific method will not be repeated here. For example, the preset threshold includes a threshold of confidence probability, a threshold of the number of position coordinates, and a preset range of position coordinates. At this time, the terminal device can output the quality inspection result of the object to be detected as failed through the comparison of the above three exemplary schemes as long as any one of them meets the condition of failing the quality inspection. Alternatively, the weights of the results of the three comparison schemes are set, and the final quality inspection result is calculated according to the weights.

[0105] In the technical solution provided in this embodiment, two-dimensional target detection is combined with photometric stereo detection, so that both surface defects of objects and deep defects of scratches on objects can be detected, thereby improving the accuracy of object detection.

[0106] It can be understood that after obtaining the second detection result in step 602, the terminal device can also calculate the estimated depth of the depth defect by using the position coordinates in the second detection result and the second image. The specific process can be as follows: Fig.11 As shown:

[0107] Steps 1101 to 1102 are the same as steps 601 to 602 above, and will not be described in detail here.

[0108] 1103. Calculate a relative depth set of each pixel point within the position coordinate indicated by the second detection result according to the position coordinate indicated by the second detection result and the color normal vector image using the path integral principle.

[0109] In this embodiment, the terminal device calculates the relative depth set of each pixel point within the position coordinates indicated by the second detection result based on the position coordinates of each defect indicated by the second detection result and the color normal vector image synthesized by photometric stereo using the path integral principle. It can be understood that each relative depth in the relative depth set is the height of the inward scratches and outward protrusions on the surface of the object to be detected, with the surface of the object to be detected as the coordinate plane.

[0110] Specifically, the process of calculating the relative depth by the terminal device may be as follows:

[0111] First, the height field is represented by z = f(x, y), and its schematic diagram can be shown as follows Fig.12 As shown, the (a, b) is used to indicate the coordinate value of a pixel point in the target bounding box indicated by the position coordinate, and the (x, y) is used to indicate the coordinate value of another pixel point in the target bounding box indicated by the position coordinate relative to the pixel point (a, b). Fig.12 It can be seen that according to the path integral principle, formula a is obtained: That is, the height difference between the two points (a, b) and (x, y) is equal to the path integral of the directional derivative. Because the normal vector field is derived from the height field, the above integral is independent of the path and can be expressed as Fig.13 As shown in the figure, a special path is selected such as (a,b)→(x,b)→(x,y). At this time, the path integral formula can be expressed as formula b: According to Fig.13 The auxiliary point (s, y) in can get formula c: Therefore, by combining formula a, formula b and formula c, we can get formula d:

[0112]

[0113] Then according to the relationship between gradient and normal:

[0114]

[0115] You can get:

[0116]

[0117] Convert the variable upper limit integral to an indefinite integral:

[0118]

[0119] Discretize the above integral on the image:

[0120]

[0121] That is, through an exemplary calculation process of the above path integral, the height difference between the two points (a, b) and (x, y) can be obtained, thereby obtaining the relative depth of the corresponding pixel points in each position coordinate in the second detection result, and then obtaining the relative depth set. It can be understood that the n x 、n y and n z A representation vector used to represent the normal vector corresponding to the pixel point.

[0122] In an exemplary embodiment, Fig.14 The figure shows a schematic diagram of depth estimation of needle scratches on the surface of a flat object. After the defect detection depth model detects the bounding box of the local needle scratch, it only has the position coordinates and confidence probability. We can use the above algorithm to estimate the relative depth. Assuming that the relative depth of the leftmost side of the bounding box is 0, the maximum relative depth is calculated by the normal vector combined with the above path integral formula to obtain the desired output value.

[0123] 1104. Select the maximum relative depth in the relative depth set as the estimated depth of the defect indicated by the second detection result.

[0124] The terminal device selects the maximum relative depth in the relative depth set as the estimated depth of the defect indicated by the second detection result. For example, the relative depth set detected in the target bounding box indicated by position coordinate 4 (a4, b4, c4, d4) is: 0.2 mm, 0.3 mm, 0.4 mm, 0.5 mm, 0.1 mm. The terminal device can select 0.5 mm as the estimated depth in the target bounding box.

[0125] 1105. Input the first detection result, the second detection result and the estimated depth into a joint processing model, compare them with a preset threshold, and output a quality inspection result of the object to be detected. The preset threshold includes a threshold for estimated depth and a threshold for confidence probability, a threshold for the number of position coordinates, and at least one of a preset range for position coordinates.

[0126] The terminal device may input the first detection result, the second detection result and the estimated depth into the joint processing model for comprehensive calculation to obtain the quality inspection result of the object to be detected. Specifically, the terminal device may compare the position coordinates of each defect in the first detection result with its corresponding confidence probability and a preset threshold, and at the same time compare the position coordinates of each defect in the second detection result and the corresponding confidence probability and the estimated depth corresponding to each position coordinate in the second detection result with the preset threshold.

[0127] In this embodiment, the preset threshold includes a threshold for the confidence probability of the first detection result, that is, to determine whether the possibility of the existence of defects in the object to be detected exceeds the threshold; the preset threshold also includes a threshold for the confidence probability of the second detection result, that is, to determine whether the possibility of the existence of defects in the object to be detected exceeds the threshold, on this basis, the preset threshold also includes a threshold for the estimated depth, that is, when confirming that the object to be detected has defects, to determine whether the scratch depth or protrusion height of the defect exceeds the threshold; the preset threshold also includes a threshold for the number of position coordinates in the first detection result and the second detection result, that is, to determine whether the defects that may exist in the object to be detected exceed the threshold; the preset threshold also includes a threshold for the position coordinates of each defect in the first detection result and the second detection result, that is, to determine whether the position where the defect may exist in the object to be detected is in an important part. It can be understood that the preset threshold includes at least one of the threshold for the confidence probability, the threshold for the number of position coordinates, and the preset range of the position coordinates.

[0128] In an exemplary solution, the preset threshold includes the threshold of the confidence probability and the threshold of the estimated depth, and the terminal device determines the quality inspection result of the object to be inspected according to the confidence probability of each defect in the first detection result and the confidence probability of each defect in the second detection result. That is, when the confidence probability indicated by the first detection result is greater than the first threshold or the confidence probability indicated by the second detection result is greater than the second threshold and the estimated depth is greater than the third threshold, the quality inspection result of the target work is output as quality inspection failure; when the confidence probability indicated by the first detection result is less than or equal to the first threshold, the confidence probability indicated by the second detection result is less than or equal to the second threshold and the estimated depth is less than or equal to the third threshold, the quality inspection result of the target work is output as quality inspection passed. It can be understood that when the first detection result and the second detection result include multiple defects and correspond to multiple confidence probabilities, as long as there is a defect with a confidence probability greater than the preset threshold, the quality inspection result of the object to be inspected can be determined as not passed. In an exemplary solution, the first detection result may include the following information: position coordinate 1 (a1, b1, c1, d1), the corresponding confidence probability is 0.5; position coordinate 2 (a2, b2, c2, d2), the corresponding confidence probability is 0.3; position coordinate 3 (a3, b3, c3, d3), the corresponding confidence probability is 0.8. That is, the terminal device detects that there are 3 surface defects of the object to be detected. The second detection result may include the following information: position coordinate 4 (a4, b4, c4, d4), the corresponding confidence probability is 0.3, and the estimated depth is 0.1 mm; position coordinate 5 (a5, b5, c5, d5), the corresponding confidence probability is 0.6, and the estimated depth is 0.3 mm; position coordinate 6 (a6, b6, c6, d6), the corresponding confidence probability is 0.8, and the estimated depth is 0.5 mm. That is, the terminal device detects that the object to be inspected has 3 depth defects. The confidence probability threshold for the surface defect is 0.5, and the confidence probability threshold for the depth defect is 0.3, so the terminal device confirms that the quality inspection result of the object to be inspected is that the quality inspection fails.

[0129] In another exemplary embodiment, the preset threshold includes a threshold of the number of position coordinates, and the terminal device determines the quality inspection result of the object to be inspected according to the number of each position coordinate in the first detection result and the number of each position coordinate in the second detection result. That is, when the sum of the number of each position coordinate indicated by the first detection result and the number of each position coordinate indicated by the second detection result is greater than the fourth threshold, the quality inspection result of the target work is output as quality inspection failure; when the sum of the number of each position coordinate indicated by the first detection result and the number of each position coordinate indicated by the second detection result is less than or equal to the fourth threshold, the quality inspection result of the target work is output as quality inspection passed. For example, the first detection result may include the following information: position coordinate 1 (a1, b1, c1, d1), the corresponding confidence probability is 0.5; position coordinate 2 (a2, b2, c2, d2), the corresponding confidence probability is 0.3; position coordinate 3 (a3, b3, c3, d3), the corresponding confidence probability is 0.8. That is, the terminal device detects that there are 3 surface defect targets of the object to be inspected. The second detection result may include the following information: position coordinate 4 (a4, b4, c4, d4), the corresponding confidence probability is 0.3; position coordinate 5 (a5, b5, c5, d5), the corresponding confidence probability is 0.6; position coordinate 6 (a6, b6, c6, d6), the corresponding confidence probability is 0.8. That is, the terminal device detects that there are 3 deep defect targets of the object to be detected, and the total number of defect targets detected by the terminal device for the object to be detected is 6. The threshold value for the number of position coordinates is set to 7, and the terminal device confirms that the quality inspection result of the object to be detected is passed.

[0130] In another exemplary embodiment, the preset threshold includes a threshold of the position coordinates, and the terminal device determines the quality inspection result of the object to be detected according to each position coordinate in the first detection result and each position coordinate in the second detection result. That is, when the first detection result indicates that each position coordinate and the second detection result indicate that each position coordinate are located in an important area (i.e., a preset range) of the object to be detected, the quality inspection result of the target work is output as quality inspection failure; when the first detection result indicates that each position coordinate and the second detection result indicate that each position coordinate are not in an important area (i.e., a preset range) of the object to be detected, the quality inspection result of the target work is output as quality inspection passed. For example, the first detection result may include the following information: position coordinate 1 (a1, b1, c1, d1), the corresponding confidence probability is 0.5; position coordinate 2 (a2, b2, c2, d2), the corresponding confidence probability is 0.3; position coordinate 3 (a3, b3, c3, d3), the corresponding confidence probability is 0.8. The second detection result may include the following information: position coordinate 4 (a4, b4, c4, d4), the corresponding confidence probability is 0.3; position coordinate 5 (a5, b5, c5, d5), the corresponding confidence probability is 0.6; position coordinate 6 (a6, b6, c6, d6), the corresponding confidence probability is 0.8. The important area of ​​the object to be detected is (A, B, C, D), where the position coordinate 1 is located in the important area, and the position coordinate 4 is located in the important area, then the terminal device confirms that the quality inspection result of the object to be detected is that the quality inspection fails.

[0131] It is understandable that the terminal device can implement any combination of the above three comparison schemes when comparing the first detection result and the second detection result with the preset threshold, as long as accurate detection of the object to be detected can be achieved, and the specific method will not be repeated here. For example, the preset threshold includes a threshold of confidence probability, a threshold of the number of position coordinates, and a preset range of position coordinates. At this time, the terminal device can output the quality inspection result of the object to be detected as failed through the comparison of the above three exemplary schemes as long as any one of them meets the condition of failing the quality inspection. Alternatively, the weights of the results of the three comparison schemes are set, and the final quality inspection result is calculated according to the weights.

[0132] In the technical solution provided in this embodiment, two-dimensional target detection is combined with photometric stereo detection, so that both surface defects of objects and depth defects of scratches on objects can be detected, thereby improving the accuracy of object detection. After the depth defect is detected by the photometric stereo target detection model, the estimated depth of the depth defect can also be estimated, and the input of the joint processing model is increased, that is, the estimated depth is added to the first detection result and the second detection result, and then the quality inspection result is determined based on these three inputs. In this way, the quality inspection result of the object is judged using the defect detection results of multiple dimensions, making the quality inspection result of the object more accurate. At the same time, when the relative depth of each pixel point within the position coordinates detected in the photometric stereo target detection model is calculated using the path integral principle, the detection area is small, thereby reducing the computational complexity of calculating the estimated depth.

[0133] The object detection device in this application is described in detail below. Fig.15 , Fig.15 This is a schematic diagram of an embodiment of an object detection device in an embodiment of the present application. The object detection device 20 includes:

[0134] An acquisition module 201 is used to acquire a first image and a second image of the object to be detected, wherein the first image is a two-dimensional image, and the second image at least includes a color normal vector image synthesized by photometric stereo;

[0135] The processing module 202 is used to input the first image into a two-dimensional object detection model to output a first detection result, and input the second image into a photometric stereo object detection model to output a second detection result, wherein the first detection result is used to indicate the position coordinates and confidence probability of each defect detected from the first image, and the second detection result is used to indicate the position coordinates and confidence probability of each defect detected from the second image;

[0136] The output module 203 is used to input the first detection result and the second detection result into a joint processing model for comparison with a preset threshold to output the quality inspection result of the object to be detected. The preset threshold includes at least one of a confidence probability threshold, a position coordinate quantity threshold and a preset range of position coordinates.

[0137] In an embodiment of the present application, an object detection device is provided. By using the above device, two-dimensional target detection is combined with photometric stereo detection, so that both surface defects of objects and deep defects of scratches on objects can be detected, thereby improving the accuracy of object detection.

[0138] Optionally, in the above Fig.15 Based on the corresponding embodiment, in another embodiment of the object detection device 20 provided in the embodiment of the present application, as Fig.16As shown, the object detection device also includes an estimation module 204, which is used to calculate the relative depth set of each pixel point within the position coordinates indicated by the second detection result according to the position coordinates indicated by the second detection result and the color normal vector image using the path integral principle; and select the maximum relative depth in the relative depth set as the estimated depth of the defect indicated by the second detection result.

[0139] In an embodiment of the present application, an object detection device is provided. With the above device, after the depth defect is detected by the photometric stereo target detection model, the estimated depth of the depth defect can also be estimated, and the input of the joint processing model is increased, so that the quality inspection result of the object is judged using the defect detection results of multiple dimensions, making the quality inspection result of the object more accurate. At the same time, when the relative depth of each pixel point within the position coordinate detected in the photometric stereo target detection model is calculated using the path integral principle, the detection area is small, thereby reducing the computational complexity of calculating the estimated depth.

[0140] Optionally, in the above Fig.16 On the basis of the corresponding embodiment, in another embodiment of the object detection device 20 provided in the embodiment of the present application, the output module 203 is specifically used to compare the first detection result, the second detection result and the estimated depth input joint processing model with a preset threshold to output the quality inspection result of the object to be detected.

[0141] In an embodiment of the present application, an object detection device is provided. The above device is used to increase the input of the joint processing model, that is, the estimated depth is added to the first detection result and the second detection result, and then the quality inspection result is determined based on these three inputs, so that the quality inspection result of the object is judged using the defect detection results of multiple dimensions, so that the quality inspection result of the object is more accurate.

[0142] Optionally, in the above Fig.16 On the basis of the corresponding embodiment, in another embodiment of the object detection device 20 provided in the embodiment of the present application, the input module 203 is specifically used to output the quality inspection result of the target work as quality inspection failure when the confidence probability indicated by the first detection result is greater than the first threshold or the confidence probability indicated by the second detection result is greater than the second threshold and the estimated depth is greater than the third threshold;

[0143] When the confidence probability indicated by the first detection result is less than or equal to the first threshold, the confidence probability indicated by the second detection result is less than or equal to the second threshold, and the estimated depth is less than or equal to the third threshold, the quality inspection result of the target work is output as quality inspection passed.

[0144] In an embodiment of the present application, an object detection device is provided. The above device is used to increase the input of the joint processing model, that is, the estimated depth is added to the first detection result and the second detection result, and then the quality inspection result is determined based on these three inputs, so that the quality inspection result of the object is judged using the defect detection results of multiple dimensions, so that the quality inspection result of the object is more accurate.

[0145] Optionally, in the above Fig.15 Or on the basis of the embodiment corresponding to 16, in another embodiment of the object detection device 20 provided in the embodiment of the present application, the acquisition module 201 is specifically used to acquire an image set of the object to be detected under different lighting conditions through a target lens and at least three light sources, and the at least three light sources are light sources located at different positions relative to the object to be detected;

[0146] Any image in the image set is selected as the first image, and images generated when the at least three light sources illuminate the object to be detected individually are selected to synthesize the second image through photometric stereo.

[0147] In an embodiment of the present application, an object detection device is provided. Using the above device, a set of images is acquired through multiple light sources in different directions and different lighting conditions for the object to be detected, and then the corresponding images are selected to generate a first image and a second image. In this way, when the lens is fixed, the acquired images have correlation, thereby realizing the function of photometric stereo synthesis of color normal vector images.

[0148] Optionally, in the above Fig.15 Or on the basis of the embodiment corresponding to 16, in another embodiment of the object detection device 20 provided in the embodiment of the present application, the acquisition module 201 is specifically used to synthesize the second image by photometric stereo based on the images generated when the at least three light sources separately illuminate the object to be detected, and the reflection direction of the at least three light sources, wherein the incident direction is determined by light source calibration.

[0149] In an embodiment of the present application, an object detection device is provided. Using the above device, a set of images is acquired through multiple light sources in different directions and different lighting conditions for the object to be detected, and then the corresponding images are selected to generate a first image and a second image. In this way, when the lens is fixed, the acquired images have correlation, thereby realizing the function of photometric stereo synthesis of color normal vector images.

[0150] Optionally, in the above Fig.15 On the basis of the embodiment corresponding to 16, in another embodiment of the object detection device 20 provided in the embodiment of the present application, when the number of the light sources is 4, the acquisition module 201 is specifically used for

[0151] When the first light source irradiates the object to be detected, a third image of the object to be detected is generated through the target lens;

[0152] When the second light source illuminates the object to be detected, a fourth image of the object to be detected is generated through the target lens;

[0153] When the third light source illuminates the object to be detected, a fifth image of the object to be detected is generated through the target lens;

[0154] When the fourth light source illuminates the object to be detected, a sixth image of the object to be detected is generated through the target lens;

[0155] When the first light source, the second light source, the third light source and the fourth light source all illuminate the object to be detected, a seventh image of the object to be detected is generated through the target lens; wherein the image set includes the third image, the fourth image, the fifth image, the sixth image and the seventh image;

[0156] The first light source, the second light source, the third light source and the fourth light source are light sources located at different positions relative to the object to be detected;

[0157] The acquisition module 201 is specifically configured to select the seventh image as the first image;

[0158] The third image, the fourth image, the fifth image and the sixth image are synthesized into the second image through photometric stereo.

[0159] In an embodiment of the present application, an object detection device is provided. Using the above device, a set of images is acquired through multiple light sources in different directions and different lighting conditions for the object to be detected, and then the corresponding images are selected to generate a first image and a second image. In this way, when the lens is fixed, the acquired images have correlation, thereby realizing the function of photometric stereo synthesis of color normal vector images.

[0160] Optionally, in the above Fig.15 Or on the basis of the embodiment corresponding to 16, in another embodiment of the object detection device 20 provided in the embodiment of the present application, the target lens is a telecentric lens or a telephoto lens;

[0161] The at least three light sources are telecentric illumination light sources with uniform intensity or long-distance point light sources.

[0162] In an embodiment of the present application, an object detection device is provided. By using the above device, restrictions on the target lens and light source are increased, so that the accuracy of photometric stereo synthesis of color normal vector images can be more effectively improved.

[0163] Optionally, in the above Fig.15On the basis of the corresponding embodiment, in another embodiment of the object detection device 20 provided in the embodiment of the present application, the second image also includes a reflectance grayscale image synthesized by photometric stereo.

[0164] In an embodiment of the present application, an object detection device is provided. By using the above device, a reflectivity grayscale image is added to the second image, thereby improving the accuracy of photometric stereo target detection.

[0165] Optionally, in the above Fig.15 On the basis of the corresponding embodiment, in another embodiment of the object detection device 20 provided in the embodiment of the present application, the acquisition module is further used to acquire a first training sample set and a second training sample set, an initial two-dimensional target detection model and an initial photometric stereo target detection model, wherein the first training sample set annotates surface defects and the second training sample set annotates depth defects;

[0166] In another embodiment, if Fig.17 The object detection device shown also includes a training module 205, which is used to use the first training sample set to perform supervised training on the initial two-dimensional target detection model to obtain the two-dimensional target detection model, and use the second training sample set to perform supervised training on the initial photometric stereo target detection model to obtain the photometric stereo target detection model.

[0167] In an embodiment of the present application, an object detection device is provided. The above device is used in a supervised training mode, and the training process of the two-dimensional object detection model and the photometric stereo object detection model are parallel, so that the two-dimensional object detection model and the photometric stereo object detection model have good generalization ability.

[0168] Optionally, in the above Fig.15 On the basis of the corresponding embodiment, in another embodiment of the object detection device 20 provided in the embodiment of the present application,

[0169] The initial two-dimensional target detection model is a cascaded convolutional neural network for region detection (Cascade Region-based Convolutional Neural Networks, Cascade RCNN), a fast convolutional neural network for region detection (Faster Region-based Convolutional Neural Networks, Faster RCNN) or a self-attention model for target detection (DEtection Transformer, DETR);

[0170] The initial photometric stereo object detection model is Cascade RCNN, Faster RCNN or DETR.

[0171] In an embodiment of the present application, an object detection device is provided. By using the above device, the model architecture of the two-dimensional target detection model and the photometric stereo target detection model is limited, so that the two-dimensional target detection model and the photometric stereo target detection model can use different targeted neural networks, making the overall target detection model more flexible.

[0172] Optionally, in the above Fig.15 On the basis of the corresponding embodiment, in another embodiment of the object detection device 20 provided in the embodiment of the present application, the feature extraction backbone network in the initial two-dimensional target detection model and the initial photometric stereo target detection model is a high-resolution network (High-Resolution netV2P, HRNetV2P) with a feature pyramid, a residual neural network (Residual Neural Network, ResNet) or a hierarchical visual self-attention model (Swin Transformer) based on a moving window.

[0173] In an embodiment of the present application, an object detection device is provided. By using the above device, the types of feature extraction backbone networks of the two-dimensional target detection model and the photometric stereo target detection model are limited, so that the two-dimensional target detection model and the photometric stereo target detection model can use different targeted neural networks, making the overall target detection model more flexible.

[0174] The object detection device provided in this application can be used in a server, see Fig.18 , Fig.18 : is a schematic diagram of a server structure provided in an embodiment of the present application. The server 300 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 322 (for example, one or more processors) and memory 332, and one or more storage media 330 (for example, one or more mass storage devices) storing application programs 342 or data 344. Among them, the memory 332 and the storage medium 330 can be short-term storage or permanent storage. The program stored in the storage medium 330 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the central processing unit 322 can be configured to communicate with the storage medium 330 to execute a series of instruction operations in the storage medium 330 on the server 300.

[0175] The server 300 may also include one or more power supplies 326, one or more wired or wireless network interfaces 350, one or more input and output interfaces 358, and / or one or more operating systems 341, such as Windows Server 2000. TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM etc.

[0176] The steps performed by the server in the above embodiment can be based on the Fig.18 The server structure shown.

[0177] The object detection device provided in this application can be used in terminal equipment. Fig.19 For the sake of convenience, only the parts related to the embodiments of the present application are shown. For specific technical details not disclosed, please refer to the method part of the embodiments of the present application. In the embodiments of the present application, the terminal device is a laptop computer as an example for explanation:

[0178] Fig.19 FIG. 1 is a block diagram of a partial structure of a laptop computer related to a terminal device provided in an embodiment of the present application. Fig.19 The laptop computer includes components such as a radio frequency (RF) circuit 410, a memory 420, an input unit 430, a display unit 440, a sensor 450, an audio circuit 460, a wireless fidelity (WiFi) module 470, a processor 480, and a power supply 490. Those skilled in the art will appreciate that Fig.19 The notebook computer structure shown in the figure does not constitute a limitation on the notebook computer, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0179] Combine the following Fig.19 A detailed introduction to the various components of a laptop computer:

[0180] The RF circuit 410 can be used for receiving and sending signals during information transmission or calls. In particular, after receiving the downlink information of the router, it is sent to the processor 480 for processing; in addition, the designed uplink data is sent to the router. Generally, the RF circuit 410 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, etc. In addition, the RF circuit 410 can also communicate with the network and other devices through wireless communication. The above wireless communication can use any communication standard or protocol, including but not limited to the global system of mobile communication (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), long term evolution (LTE), email, short messaging service (SMS), etc.

[0181] The memory 420 can be used to store software programs and modules. The processor 480 executes various functional applications and data processing of the notebook computer by running the software programs and modules stored in the memory 420. The memory 420 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the notebook computer (such as audio data, video data, etc.), etc. In addition, the memory 420 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0182] The input unit 430 can be used to receive input digital or character information, and to generate key signal input related to the user settings and function control of the laptop computer. Specifically, the input unit 430 may include a touch panel 431 and other input devices 432. The touch panel 431, also known as a touch screen, can collect the user's touch operation on or near it (such as the user's operation on the touch panel 431 or near the touch panel 431 using any suitable object or accessory such as a finger, stylus, etc.), and drive the corresponding connection device according to a pre-set program. Optionally, the touch panel 431 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into the coordinates of the contact point, and then sends it to the processor 480, and can receive and execute the command sent by the processor 480. In addition, the touch panel 431 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic waves. In addition to the touch panel 431, the input unit 430 may also include other input devices 432. Specifically, other input devices 432 may include but are not limited to one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, etc.

[0183] The display unit 440 may be used to display information input by the user or information provided to the user and various menus of the laptop computer. The display unit 440 may include a display panel 441. Optionally, the display panel 441 may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like. Further, the touch panel 431 may cover the display panel 441. When the touch panel 431 detects a touch operation on or near it, it is transmitted to the processor 480 to determine the type of touch event. Subsequently, the processor 480 provides a corresponding visual output on the display panel 441 according to the type of touch event. Although in Fig.19 In the embodiment, the touch panel 431 and the display panel 441 are used as two independent components to implement the input and output functions of the notebook computer, but in some embodiments, the touch panel 431 and the display panel 441 can be integrated to implement the input and output functions of the notebook computer.

[0184] The laptop computer may also include at least one sensor 450, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 441 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 441 and / or the backlight when the laptop computer is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the laptop computer (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors that can be configured in the laptop computer, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be repeated here.

[0185] The audio circuit 460, the speaker 461, and the microphone 462 can provide an audio interface between the user and the laptop computer. The audio circuit 460 can transmit the received audio data to the speaker 461 after converting the received audio data into an electrical signal, which is converted into a sound signal for output; on the other hand, the microphone 462 converts the collected sound signal into an electrical signal, which is received by the audio circuit 460 and converted into audio data, and then the audio data is output to the processor 480 for processing, and then sent to another laptop computer through the RF circuit 410, or the audio data is output to the memory 420 for further processing.

[0186] WiFi is a short-range wireless transmission technology. The laptop computer can help users send and receive emails, browse web pages, and access streaming media through the WiFi module 470. It provides users with wireless broadband Internet access. Fig.19 A WiFi module 470 is shown, but it is understandable that it is not an essential component of the laptop computer and can be omitted as required without changing the essence of the invention.

[0187] The processor 480 is the control center of the laptop computer. It uses various interfaces and lines to connect various parts of the entire laptop computer. It executes various functions of the laptop computer and processes data by running or executing software programs and / or modules stored in the memory 420, and calling data stored in the memory 420, so as to monitor the laptop computer as a whole. Optionally, the processor 480 may include one or more processing units; optionally, the processor 480 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 480.

[0188] The laptop computer also includes a power supply 490 (such as a battery) for supplying power to various components. Optionally, the power supply can be logically connected to the processor 480 through a power management system, thereby implementing functions such as charging, discharging, and power consumption management through the power management system.

[0189] Although not shown, the laptop computer may also include a camera, a Bluetooth module, etc., which will not be described in detail here.

[0190] The steps performed by the terminal device in the above embodiment can be based on the Fig.19 The terminal device structure shown.

[0191] A computer-readable storage medium is also provided in an embodiment of the present application. The computer-readable storage medium stores a computer program, which, when executed on a computer, enables the computer to execute the methods described in the aforementioned embodiments.

[0192] The present application also provides a computer program product including a program, which, when executed on a computer, enables the computer to execute the methods described in the aforementioned embodiments.

[0193] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0194] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0195] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0196] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0197] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0198] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An object detection method, characterized in that: include: Acquire a first image and a second image of the object to be detected, wherein the first image is a two-dimensional image, and the second image includes at least a color normal vector image synthesized by photometric stereo and a reflectance grayscale image synthesized by photometric stereo, wherein the first image is an image obtained by selecting multiple light sources to illuminate the object simultaneously, and the second image is an image obtained by performing wide stereo synthesis on images generated when at least three light sources are selected to illuminate the object to be detected separately; Input the first image into a two-dimensional object detection model to output a first detection result, and input the second image into a photometric stereo object detection model to output a second detection result, wherein the first detection result is used to indicate the position coordinates and confidence probability of each defect detected from the first image, and the second detection result is used to indicate the position coordinates and confidence probability of each defect detected from the second image; Calculating a relative depth set of each pixel point within the position coordinate indicated by the second detection result according to the position coordinate indicated by the second detection result and the color normal vector image by using the path integral principle; Selecting the maximum relative depth in the relative depth set as the estimated depth of the defect indicated by the second detection result; The first detection result, the second detection result and the estimated depth are input into a joint processing model and compared with preset thresholds to output a quality inspection result of the workpiece to be inspected, wherein the preset thresholds include a confidence probability threshold, a position coordinate quantity threshold, a position coordinate preset range and an estimated depth threshold; Among them, the model training methods include: Obtaining a first training sample set and a second training sample set corresponding to the labeled boxes, as well as an initial two-dimensional target detection model and an initial photometric stereo target detection model, wherein the first training sample set labels surface defects and the second training sample set labels depth defects; Performing detection processing on the first training sample set by using an initial two-dimensional target detection model to obtain a prediction box including object defects in the first training sample set; Determine a loss value corresponding to the prediction box according to a loss function including two factors, namely, an interval value and a distance, and perform supervised training on the initial two-dimensional object detection model based on the loss value corresponding to the prediction box to obtain the two-dimensional object detection model; Performing detection processing on the second training sample set by using an initial photometric stereo object detection model to obtain a prediction box including object defects in the second training sample set; Determine a loss value corresponding to the prediction box according to a loss function including an interval value and a distance, and perform supervised training on an initial photometric stereo object detection model based on the loss value corresponding to the prediction box to obtain the photometric stereo object detection model; Among them, the initial two-dimensional target detection model is a cascaded convolutional neural network for region detection CascadeRCNN, a fast convolutional neural network for region detection Faster RCNN or a self-attention model DETR for target detection, and the initial photometric stereo target detection model is Cascade RCNN, Faster RCNN or DETR.

2. The method according to claim 1, characterized in that The step of inputting the first detection result, the second detection result and the estimated depth into a joint processing model and comparing them with the preset threshold to output the quality inspection result of the workpiece to be detected includes: When the confidence probability indicated by the first detection result is greater than a first threshold or the confidence probability indicated by the second detection result is greater than a second threshold and the estimated depth is greater than a third threshold, outputting a quality inspection result of the target work as quality inspection failure; When the confidence probability indicated by the first detection result is less than or equal to the first threshold, the confidence probability indicated by the second detection result is less than or equal to the second threshold, and the estimated depth is less than or equal to the third threshold, the quality inspection result of the target work is output as quality inspection passed.

3. The method according to claim 1, characterized in that The acquiring of the first image and the second image of the object to be detected comprises: Acquire a set of images of the object to be detected under different lighting conditions through a target lens and at least three light sources, wherein the at least three light sources are light sources located at different positions relative to the object to be detected; Any one image in the image set is selected as the first image, and images generated when the at least three light sources illuminate the object to be detected individually are selected to synthesize the second image through photometric stereo.

4. The method according to claim 3, characterized in that The selecting the at least three light sources to individually illuminate the object to be detected and synthesizing the second image through photometric stereo synthesis comprises: The second image is synthesized by photometric stereo from images generated when the at least three light sources illuminate the object to be detected separately according to the incident directions of the at least three light sources, the surface reflectivity of the object to be detected and the reflection directions of the at least three light sources, wherein the incident direction is determined by light source calibration.

5. The method according to claim 3, characterized in that: When the number of the light sources is 4, the image set of the target working under different lighting conditions obtained through the target lens and at least three lighting light sources in different directions includes: When the first light source irradiates the object to be detected, a third image of the object to be detected is generated through the target lens; When the second light source illuminates the object to be detected, a fourth image of the object to be detected is generated through the target lens; When the third light source illuminates the object to be detected, a fifth image of the object to be detected is generated through the target lens; When the fourth light source illuminates the object to be detected, a sixth image of the object to be detected is generated through the target lens; When the first light source, the second light source, the third light source and the fourth light source all illuminate the object to be detected, a seventh image of the object to be detected is generated through the target lens; wherein the image set includes the third image, the fourth image, the fifth image, the sixth image and the seventh image; The first light source, the second light source, the third light source and the fourth light source are light sources located at different positions relative to the object to be detected; The selecting any one image in the image set as the first image, and selecting images generated when at least three light sources are individually illuminated on the object to be detected to synthesize the second image through photometric stereo synthesis includes: selecting the seventh image as the first image; The third image, the fourth image, the fifth image and the sixth image are synthesized into the second image through photometric stereo.

6. The method according to claim 3, characterized in that The target lens is a telecentric lens or a telephoto lens; The at least three light sources are telecentric illumination light sources with uniform intensity or long-distance point light sources.

7. The method according to claim 1, characterized in that The feature extraction backbone network in the initial two-dimensional target detection model and the initial photometric stereo target detection model is a high-resolution network HRNetV2P with a feature pyramid, a residual neural network ResNet, or a hierarchical visual self-attention model Swin Transformer based on a moving window.

8. An object detection device, characterized in that: include: An acquisition module, used to acquire a first image and a second image of the object to be detected, wherein the first image is a two-dimensional image, and the second image at least includes a color normal vector image synthesized by photometric stereo and a reflectance grayscale image synthesized by photometric stereo, wherein the first image is an image obtained by selecting multiple light sources to illuminate the object simultaneously, and the second image is an image obtained by performing wide stereo synthesis on images generated when at least three light sources are selected to illuminate the object to be detected separately; a processing module, configured to input the first image into a two-dimensional object detection model to output a first detection result, and input the second image into a photometric stereo object detection model to output a second detection result, wherein the first detection result is used to indicate the position coordinates and confidence probability of each defect detected from the first image, and the second detection result is used to indicate the position coordinates and confidence probability of each defect detected from the second image; an estimation module, configured to calculate a relative depth set of each pixel point within the position coordinate indicated by the second detection result according to the position coordinate indicated by the second detection result and the color normal vector image by using a path integral principle; and select a maximum relative depth in the relative depth set as an estimated depth of the defect indicated by the second detection result; An output module, used to compare the first detection result, the second detection result and the estimated depth input into a joint processing model with preset thresholds to output a quality inspection result of the workpiece to be inspected, wherein the preset thresholds include a confidence probability threshold, a position coordinate quantity threshold, a position coordinate preset range and an estimated depth threshold; A training module, used to obtain a first training sample set and a second training sample set corresponding to annotated boxes, as well as an initial two-dimensional target detection model and an initial photometric stereo target detection model, wherein the first training sample set annotates surface defects and the second training sample set annotates depth defects; the first training sample set is detected and processed by the initial two-dimensional target detection model to obtain a prediction box including object defects in the first training sample set; a loss value corresponding to the prediction box is determined according to a loss function including two factors of interval value and distance, and based on the loss value corresponding to the prediction box, the initial two-dimensional target detection model is supervisedly trained to obtain the two-dimensional target detection model; the second training sample set is detected and processed by the initial photometric stereo target detection model to obtain a prediction box including object defects in the second training sample set; the loss value corresponding to the prediction box is determined according to a loss function including two factors of interval value and distance, and based on the loss value corresponding to the prediction box, the initial photometric stereo target detection model is supervisedly trained to obtain the photometric stereo target detection model; Among them, the initial two-dimensional target detection model is a cascaded convolutional neural network for region detection CascadeRCNN, a fast convolutional neural network for region detection Faster RCNN or a self-attention model DETR for target detection, and the initial photometric stereo target detection model is Cascade RCNN, Faster RCNN or DETR.

9. The device according to claim 8, characterized in that The output module is specifically used for: When the confidence probability indicated by the first detection result is greater than a first threshold or the confidence probability indicated by the second detection result is greater than a second threshold and the estimated depth is greater than a third threshold, outputting a quality inspection result of the target work as quality inspection failure; When the confidence probability indicated by the first detection result is less than or equal to the first threshold, the confidence probability indicated by the second detection result is less than or equal to the second threshold, and the estimated depth is less than or equal to the third threshold, the quality inspection result of the target work is output as quality inspection passed.

10. The device according to claim 8, characterized in that The acquisition module is specifically used for: Acquire a set of images of the object to be detected under different lighting conditions through a target lens and at least three light sources, wherein the at least three light sources are light sources located at different positions relative to the object to be detected; Any one image in the image set is selected as the first image, and images generated when the at least three light sources illuminate the object to be detected individually are selected to synthesize the second image through photometric stereo.

11. The device according to claim 10, characterized in that The acquisition module is specifically used for: The second image is synthesized by photometric stereo from images generated when the at least three light sources illuminate the object to be detected separately according to the incident directions of the at least three light sources, the surface reflectivity of the object to be detected and the reflection directions of the at least three light sources, wherein the incident direction is determined by light source calibration.

12. The device according to claim 10, characterized in that When the number of the light sources is 4, the acquisition module is specifically used to: When the first light source irradiates the object to be detected, a third image of the object to be detected is generated through the target lens; When the second light source illuminates the object to be detected, a fourth image of the object to be detected is generated through the target lens; When the third light source illuminates the object to be detected, a fifth image of the object to be detected is generated through the target lens; When the fourth light source illuminates the object to be detected, a sixth image of the object to be detected is generated through the target lens; When the first light source, the second light source, the third light source and the fourth light source all illuminate the object to be detected, a seventh image of the object to be detected is generated through the target lens; wherein the image set includes the third image, the fourth image, the fifth image, the sixth image and the seventh image; The first light source, the second light source, the third light source and the fourth light source are light sources located at different positions relative to the object to be detected; The selecting any one image in the image set as the first image, and selecting images generated when at least three light sources are individually illuminated on the object to be detected to synthesize the second image through photometric stereo synthesis includes: selecting the seventh image as the first image; The third image, the fourth image, the fifth image and the sixth image are synthesized into the second image through photometric stereo.

13. The device according to claim 10, characterized in that The target lens is a telecentric lens or a telephoto lens; The at least three light sources are telecentric illumination light sources with uniform intensity or long-distance point light sources.

14. The device according to claim 8, characterized in that The feature extraction backbone network in the initial two-dimensional target detection model and the initial photometric stereo target detection model is a high-resolution network HRNetV2P with a feature pyramid, a residual neural network ResNet, or a hierarchical visual self-attention model SwinTransformer based on a moving window.

15. A computer device, characterized in that: include: Memory, processor, and bus system; Wherein, the memory is used to store programs; The processor is used to execute the program in the memory, and the processor is used to execute the method according to any one of claims 1 to 7 according to the instructions in the program code; The bus system is used to connect the memory and the processor so that the memory and the processor can communicate with each other.

16. A computer-readable storage medium comprising instructions, which, when executed on a computer, causes the computer to perform the method according to any one of claims 1 to 7.

17. A computer program product, characterized in that The computer program product comprises a program, and when the program is run on a computer, the computer is caused to execute the method according to any one of claims 1 to 7.

Images