Object stacking anomaly detection method, device, equipment and medium

Abstract

The invention discloses an object stacking anomaly detection method, device and equipment and a medium, and is applied to the technical field of artificial intelligence, and the method comprises the steps: obtaining depth data, radar point cloud data and at least one two-dimensional image of a target region; a plurality of target objects are stacked in the target area; according to the depth data, the radar point cloud data and the at least one two-dimensional image, the number of target objects and first position information, height information, category information and spacing information of each target object are obtained, the spacing information of the target objects is spacing between the target objects and adjacent objects, and the adjacent objects are adjacent to the target objects; the first position information of the target object is position information of the target object in the target area; and according to the number of the target objects and the first position information, the height information, the category information and the spacing information of each target object, carrying out object stacking anomaly detection on the target area. According to the invention, the accuracy and efficiency of object stacking anomaly detection can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to methods, devices, equipment and media for detecting abnormal stacking of objects. Background Art

[0002] In scenes such as industrial workshops and commercial places, there are often abnormal object stacking phenomena such as illegal stacking of objects, excessive stacking of objects, and excessive stacking of objects, which not only affect normal traffic, but may also cause some safety accidents. In related technologies, manual methods are usually used to detect and handle these abnormal object stacking phenomena. However, related technologies have problems such as low detection efficiency and limited detection coverage, and are prone to detection omissions. The accuracy and efficiency of abnormal object stacking detection need to be improved. Summary of the invention

[0003] The embodiments of the present application provide a method, device, equipment and medium for detecting abnormal stacking of objects, which are used to improve the accuracy and efficiency of detecting abnormal stacking of objects.

[0004] On the one hand, an embodiment of the present application provides a method for detecting abnormal stacking of objects, comprising the following steps:

[0005] Acquire depth data, radar point cloud data and at least one two-dimensional image of a target area; the target area has a plurality of target objects stacked therein;

[0006] According to the depth data, the radar point cloud data and at least one of the two-dimensional images, obtain the number of target objects and first position information, height information, category information and spacing information of each of the target objects, wherein the spacing information of the target objects is the spacing between the target object and an adjacent object, the adjacent object is adjacent to the target object, and the first position information of the target object is the position information of the target object in the target area;

[0007] According to the number of target objects and the first position information, height information, category information and spacing information of each target object, object stacking abnormality detection is performed on the target area.

[0008] On the other hand, an embodiment of the present application provides an object stacking abnormality detection device, comprising:

[0009] An acquisition module, used to acquire depth data, radar point cloud data and at least one two-dimensional image of a target area; the target area has a plurality of target objects stacked therein;

[0010] a first processing module, configured to obtain the number of target objects and first position information, height information, category information and spacing information of each target object according to the depth data, the radar point cloud data and at least one of the two-dimensional images, wherein the spacing information of the target object is the spacing between the target object and an adjacent object, the adjacent object is adjacent to the target object, and the first position information of the target object is the position information of the target object in the target area;

[0011] The second processing module is used to perform object stacking abnormality detection on the target area according to the number of the target objects and the first position information, height information, category information and spacing information of each of the target objects.

[0012] In another aspect, an embodiment of the present application provides an electronic device, including:

[0013] at least one processor;

[0014] at least one memory for storing at least one program;

[0015] When the at least one program is executed by the at least one processor, the at least one processor implements the above-mentioned object stacking abnormality detection method.

[0016] On the other hand, an embodiment of the present application provides a computer-readable storage medium, which stores a program executable by a processor. When the program executable by the processor is executed by the processor, it is used to implement the above-mentioned object stacking abnormality detection method.

[0017] The object stacking abnormality detection method, device, equipment and medium of the embodiment of the present application first obtains the depth data, radar point cloud data and at least one two-dimensional image of the target area, wherein the target area is stacked with multiple target objects; then, according to the depth data, radar point cloud data and at least one two-dimensional image of the target area, the number of target objects and the first position information, height information, category information and spacing information of each target object are obtained, wherein the spacing information of the target object is the spacing between the target object and the adjacent object, and the adjacent object is adjacent to the target object; finally, according to the number of target objects and the first position information, height information, category information and spacing information of each target object, the target area is subjected to object stacking abnormality detection. It can be seen that the embodiment of the present application extracts the stacking characteristic data of each target object, such as the number of target objects and the first position information, height information, category information and spacing information of each target object, etc., based on the stacking characteristic data of each target object, and realizes the automatic detection of object stacking abnormality, so that the accuracy and efficiency of object stacking abnormality detection can be effectively improved.

[0018] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a flow chart of the object stacking anomaly detection method provided by the present application;

[0020] Figure 2 It is a flowchart of the processing of depth data, radar point cloud data and two-dimensional images provided by this application;

[0021] Figure 3 It is a flow chart for generating a three-dimensional point cloud model of a target object provided by the present application;

[0022] Figure 4 It is a flow chart of extracting shape features of a target object provided by the present application;

[0023] Figure 5 is another flow chart for generating a three-dimensional point cloud model of a target object provided by the present application;

[0024] Figure 6 This is a flow chart for obtaining the number of target objects and the height information and spacing information of the target objects provided by the present application;

[0025] Figure 7 is a flow chart of obtaining first position information of a target object provided by the present application;

[0026] Figure 8 is a flow chart of object stacking anomaly detection provided by the present application;

[0027] Fig. 9 is a structural diagram of the object stacking abnormality detection device provided by the present application;

[0028] Fig.10 It is a structural diagram of the electronic device provided in this application. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0030] The present application is further described below in conjunction with the accompanying drawings and specific embodiments. The described embodiments should not be regarded as limiting the present application, and all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

[0031] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0032] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0033] In scenes such as industrial workshops and commercial places, there are often abnormal stacking of objects, such as illegal stacking of objects, excessive stacking of objects, and excessive stacking of objects, which not only affect normal traffic, but may also cause some safety accidents.

[0034] For example, the aisles and work areas of industrial workshops are usually areas where workers and equipment frequently move around. Keeping the aisles and work areas unobstructed is crucial for industrial operations, and objects are usually not allowed to be stacked in these aisles and work areas. However, in actual work, objects are often temporarily or permanently stacked in these places where objects are not allowed to be stacked, which not only affects the normal passage of workers and equipment, but also causes aisle obstruction, and in serious cases, causes safety accidents such as collisions and tripping. In addition, the stacking height and number of objects are often set with corresponding safety standards. If the height of the objects stacked exceeds the preset height standard, phenomena such as objects tipping over may occur. If the number of objects stacked exceeds the preset number standard, phenomena such as floor overload may occur, all of which may lead to safety accidents.

[0035] It can be seen that in scenarios such as industrial workshops and commercial places, it is crucial to promptly detect and deal with abnormal stacking of objects such as illegal stacking, over-high stacking, and excessive stacking.

[0036] In the related art, manual methods are usually used to detect and handle abnormal object stacking. However, the related art has problems such as low detection efficiency and limited detection coverage, and is prone to detection omissions. The accuracy and efficiency of abnormal object stacking detection need to be improved.

[0037] In view of this, the embodiments of the present application provide a method, device, equipment and medium for detecting abnormal object stacking, aiming to use machine vision recognition technology to realize automatic detection of abnormal object stacking and improve the accuracy and efficiency of abnormal object stacking detection.

[0038] First, the implementation steps of the object stacking anomaly detection method provided in the embodiment of the present application will be described in detail below with reference to the accompanying drawings.

[0039] The object stacking anomaly detection method provided in the embodiment of the present application can be applied to a terminal, a server, or software running in a terminal or a server. The terminal can be a tablet computer, a laptop computer, a desktop computer, etc., but is not limited to this. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN) and big data and artificial intelligence platforms. In addition, the server can also be a node server in a blockchain network, but is not limited to this. Among them, blockchain is a new application mode of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanism, and encryption algorithm.

[0040] Reference Figure 1 , Figure 1 : is a flow chart of the object stacking anomaly detection method provided by the present application, and the object stacking anomaly detection method may include the following steps S101-S103:

[0041] S101, acquiring depth data, radar point cloud data and at least one two-dimensional image of a target area; a plurality of target objects are stacked in the target area;

[0042] S102, obtaining the number of target objects and first position information, height information, category information and spacing information of each target object according to the depth data, the radar point cloud data and the at least one two-dimensional image, wherein the spacing information of the target object is the spacing between the target object and an adjacent object, the adjacent object is adjacent to the target object, and the first position information of the target object is the position information of the target object in the target area;

[0043] S103, performing object stacking abnormality detection on the target area according to the number of target objects and the first position information, height information, category information and spacing information of each target object.

[0044] In the embodiment of the present application, first, the depth data, radar point cloud data and at least one two-dimensional image of the target area are obtained, wherein the target area is stacked with multiple target objects; then, according to the depth data, radar point cloud data and at least one two-dimensional image of the target area, the number of target objects and the first position information, height information, category information and spacing information of each target object are obtained, wherein the spacing information of the target object is the spacing between the target object and the adjacent object, and the adjacent object is adjacent to the target object; finally, according to the number of target objects and the first position information, height information, category information and spacing information of each target object, the target area is subjected to object stacking anomaly detection. It can be seen that, under the condition of fully considering multimodal data such as two-dimensional images, point cloud data and depth data, the stacking characteristic data of each target object, such as the number of target objects and the first position information, height information, category information and spacing information of each target object, are extracted, and on the basis of the stacking characteristic data of each target object, the automatic detection of object stacking anomaly is realized, so that the accuracy and efficiency of object stacking anomaly detection can be effectively improved.

[0045] In the above step S101, depth data, radar point cloud data and at least one two-dimensional image of the target area are obtained, wherein a plurality of target objects are stacked in the target area, so that stacking characteristic data of each target object can be extracted in subsequent steps.

[0046] The above-mentioned target area can be set according to actual conditions, and the embodiment of the present application does not make specific limitations on this. For example, the above-mentioned target area can be a local area of ​​a warehouse; or, the above-mentioned target area can be an aisle of an industrial workshop, but is not limited to this.

[0047] The target objects mentioned above refer to objects piled in the target area.

[0048] The number of the above-mentioned target objects can be set according to actual conditions, and the embodiments of the present application do not specifically limit this.

[0049] The radar point cloud data of the target area may include radar point cloud data of each target object.

[0050] The depth data of the target area may include depth data of each target object.

[0051] The above-mentioned acquisition of depth data, radar point cloud data and at least one two-dimensional image of the target area may include performing image acquisition on the target area through at least one image acquisition module to obtain at least one two-dimensional image, performing an environmental scan of the target area through at least one laser radar to obtain radar point cloud data of each target object as the radar point cloud data of the target area, and performing an environmental scan of the target area through at least one depth acquisition module to obtain depth data of each target object as the depth data of the target area, but is not limited to this.

[0052] The image acquisition module may be a camera based on a complementary metal oxide semiconductor (CMOS); or, the image acquisition module may be a camera based on a charge coupled device (CCD), but is not limited thereto.

[0053] The depth acquisition module may be a depth camera, but is not limited thereto.

[0054] The number and positions of the above-mentioned image acquisition modules, laser radars and depth acquisition modules can be set according to actual conditions, and the embodiments of the present application do not make specific limitations on this.

[0055] It should be noted that if at least two image acquisition modules are provided in the target area, the positions of the image acquisition modules in the target area are different, so as to acquire two-dimensional images of the target area from different viewing angles.

[0056] In the above step S102, after obtaining the depth data, radar point cloud data and at least one two-dimensional image of the target area, the stacking characteristic data of each target object is extracted, such as the number of target objects and the first position information, height information, category information and spacing information of each target object, while fully considering multimodal data such as two-dimensional images, point cloud data and depth data.

[0057] The number of target objects mentioned above refers to the total number of target objects piled in the target area.

[0058] The first position information of the target object refers to the position information of the target object in the target area.

[0059] The distance information of the target object refers to the distance between the target object and an adjacent object, and the adjacent object refers to a target object adjacent to the target object.

[0060] The above-mentioned obtaining the number of target objects and the first position information, height information, category information and spacing information of each target object based on the depth data, radar point cloud data and at least one two-dimensional image may include processing the depth data to obtain the depth value of each target object, calculating the height information of each target object through the depth value of each target object, and using the radar point cloud data to locate each target object to obtain the first position information of each target object, calculating the spacing information of each target object based on the first position information of each target object, and performing target recognition on at least one two-dimensional image to obtain the number of target objects and the category information of each target object, but is not limited to this.

[0061] In the above step S103, based on the stacking characteristic data of each target object, object stacking anomaly detection is performed on the target area, thereby realizing automatic detection of object stacking anomaly.

[0062] The above-mentioned object stacking anomaly detection in the target area based on the number of target objects and the first position information, height information, category information and spacing information of each target object may include performing object stacking anomaly detection in the target area based on the number of target objects and the first position information, height information, category information and spacing information of each target object in combination with a machine learning method, but is not limited to this.

[0063] The above-mentioned machine learning method can be set according to actual conditions, and the embodiments of the present application do not make any specific limitations on this.

[0064] For example, the above-mentioned machine learning method may be a gradient boosting machine; or, the above-mentioned machine learning method may be a random forest or logistic regression, but is not limited thereto.

[0065] The above steps will be further described below.

[0066] In some embodiments, reference Figure 2 In the above step S102, the process of obtaining the number of target objects and the first position information, height information, category information and spacing information of each target object according to the depth data, radar point cloud data and at least one two-dimensional image may include the following steps S1021-S1024:

[0067] S1021, fusing the depth data, the radar point cloud data, and at least one two-dimensional image to obtain a three-dimensional point cloud model of each target object; the three-dimensional point cloud model of the target object includes a plurality of three-dimensional point cloud data, and the three-dimensional point cloud model of the target object is used to characterize the three-dimensional position information and shape characteristics of the target object;

[0068] S1022, obtaining the number of target objects and height information and spacing information of each target object according to the three-dimensional point cloud model of each target object;

[0069] S1023, performing object detection on at least one two-dimensional image to obtain second position information and category information of each target object, where the second position information of the target object is position information of the target object in the two-dimensional image;

[0070] S1024: Obtain first position information of each target object according to the second position information and the three-dimensional position information of each target object.

[0071] In this embodiment, for the extraction of the number of target objects and the height information and spacing information of each target object, the depth data, radar point cloud data and at least one two-dimensional image are first fused and processed to construct a three-dimensional point cloud model of each target object. The three-dimensional point cloud model is mainly used to characterize the three-dimensional position information and shape characteristics of the target object, which may include a number of three-dimensional point cloud data. The three-dimensional point cloud model of each target object is then used to determine the number of target objects and the height information and spacing information of each target object. For the extraction of the first position information and category information of each target object, object detection is first performed on at least one two-dimensional image to obtain the second position information and category information of each target object. The second position information and three-dimensional position information of each target object are then used to determine the first position information of each target object. It can be seen that by fusing the depth data, radar point cloud data and two-dimensional images, a three-dimensional point cloud model of each target object is generated, and then these three-dimensional point cloud models are used to extract the number of target objects and the height information and spacing information of each target object. In addition, the position information and category information of each target object are preliminarily determined by performing object detection on the two-dimensional image, and then the three-dimensional position information in each three-dimensional point cloud model is used to correct the position information of each target object obtained by the object detection, so as to obtain the first position information of each target object. In this way, the accuracy of the stacking characteristic data of each target object can be effectively improved, so that in the subsequent steps, based on the stacking characteristic data of each target object, automatic detection of abnormal object stacking can be realized.

[0072] The above-mentioned three-dimensional point cloud model may include a plurality of three-dimensional point cloud data, which together describe the three-dimensional position information and shape characteristics of the target object, that is, the three-dimensional point cloud model can be used to characterize the three-dimensional position information and shape characteristics of the target object.

[0073] The above-mentioned shape features can be set according to actual conditions, and this embodiment does not specifically limit this. For example, the above-mentioned shape features can be edge features, contour features, height features or width features, but are not limited thereto.

[0074] The above-mentioned fusion processing of the depth data, radar point cloud data and at least one two-dimensional image to obtain the three-dimensional point cloud model of each target object may include converting the depth data and at least one two-dimensional image into corresponding point cloud data, and then using a three-dimensional reconstruction algorithm to process the radar point cloud data and the point cloud data obtained by the conversion processing to generate a three-dimensional point cloud model of each target object, but is not limited to this.

[0075] The above three-dimensional reconstruction algorithm can be set according to actual conditions, and this embodiment does not make any specific limitation to this.

[0076] For example, the above three-dimensional reconstruction algorithm may be a voxel-based three-dimensional reconstruction algorithm; or, the above three-dimensional reconstruction algorithm may be a triangulation-based three-dimensional reconstruction algorithm, but is not limited thereto.

[0077] The above-mentioned obtaining the number of target objects and the height information and spacing information of each target object based on the three-dimensional point cloud model of each target object may include analyzing and processing the three-dimensional point cloud model of each target object through a point cloud model analysis tool to obtain the number of target objects and the height information and spacing information of each target object, but is not limited to this.

[0078] The above-mentioned point cloud model analysis tool can be set according to actual conditions, and this embodiment does not make any specific limitation to this.

[0079] For example, the point cloud model analysis tool may be Lidar Studio or CloudCompare, but is not limited thereto.

[0080] The second position information of the target object refers to the position information of the target object in the two-dimensional image.

[0081] The above-mentioned object detection on at least one two-dimensional image to obtain the second position information and category information of each target object may include inputting at least one two-dimensional image into a preset first neural network model, and outputting the second position information and category information of each target object through the preset first neural network model, but is not limited to this.

[0082] The above-mentioned first neural network model can be set according to actual conditions, and this embodiment does not make any specific limitation to this.

[0083] For example, the first neural network model may be a convolutional neural network model; or, the first neural network model may be Faster-RCNN, but is not limited thereto.

[0084] The above step S1021 and the above step S1023 may be performed in parallel, for example, the above step S1021 and the above step S1023 are performed at the same time.

[0085] Alternatively, the above step S1021 and the above step S1023 may be performed in series, for example, the above step S1021 is performed first, and then the above step S1023 is performed; or, the above step S1023 is performed first, and then the above step S1021 is performed.

[0086] The above step S1022 and the above step S1023 may be performed in parallel, for example, the above step S1022 and the above step S1023 are performed at the same time.

[0087] Alternatively, the above step S1022 and the above step S1023 may be performed in series, for example, the above step S1022 is performed first, and then the above step S1023 is performed; or, the above step S1023 is performed first, and then the above step S1022 is performed.

[0088] The above-mentioned method of obtaining the first position information of each target object based on the second position information and the three-dimensional position information of each target object may include, for each target object, converting the second position information of the target object into position information in a three-dimensional world coordinate system, and then calculating the average of the position information of the target object in the three-dimensional world coordinate system and the three-dimensional position information in the three-dimensional point cloud model as the first position information of the target object, but is not limited to this.

[0089] In some embodiments, reference Figure 3 In the above step S1021, the process of fusing the depth data, the radar point cloud data and at least one two-dimensional image to obtain the three-dimensional point cloud model of each target object may include the following steps A101-A105:

[0090] A101, extracting features from at least one two-dimensional image to obtain shape features of each target object, where the shape features include at least one of edge features or contour features;

[0091] A102, performing target recognition processing on at least one two-dimensional image to obtain third position information of each target object, where the third position information of the target object includes pixel coordinates of the target object in each two-dimensional image;

[0092] A103, converting the third position information of each target object to obtain fourth position information of each target object; the fourth position information of the target object is the position information of the target object in the world coordinate system;

[0093] A104, performing point cloud conversion processing on the depth data to obtain depth point cloud data;

[0094] A105, fusing the depth point cloud data, the radar point cloud data, the shape features of each target object and the fourth position information of each target object to obtain a three-dimensional point cloud model of each target object.

[0095] In this embodiment, first, feature extraction is performed on at least one two-dimensional image to extract the shape features of each target object, the shape features include at least one of edge features or contour features, and target recognition processing is performed on at least one two-dimensional image to obtain the pixel coordinates of each target object in each two-dimensional image, that is, the third position information of each target object, and then the position information of each target object in each two-dimensional image is converted into the position information of each target object in the world coordinate system, that is, the fourth position information of each target object, and point cloud conversion processing is performed on the depth data to convert the depth data into data in the form of point cloud, so as to obtain depth point cloud data; then, the depth point cloud data, radar point cloud data, the shape features of each target object and the fourth position information of each target object are fused to obtain a three-dimensional point cloud model of each target object. In this way, under the condition of fully considering the shape features and spatial positions of the target objects and the multi-modal features such as the depth point cloud data and the radar point cloud data of the target area, the three-dimensional point cloud model of each target object is generated by registering and fusing the multi-view point cloud data, which can effectively improve the generation accuracy of the three-dimensional point cloud model, thereby improving the extraction accuracy of the stacking feature data of each target object.

[0096] The above-mentioned shape feature may include at least one of an edge feature or a contour feature.

[0097] The above-mentioned feature extraction of at least one two-dimensional image to obtain the shape features of each target object may include performing object detection on each two-dimensional image to obtain the target object in each two-dimensional image, and then performing feature extraction on the target object in each two-dimensional image to obtain the shape features of the target object in each two-dimensional image. For each target object, the shape features obtained by feature extraction of different two-dimensional images are fused into the shape features of the target object, but is not limited to this.

[0098] The third position information of the target object may include pixel coordinates of the target object in each two-dimensional image, and the third position information of the target object reflects the pixel coordinates of the target object at different viewing angles.

[0099] The above-mentioned target recognition processing of at least one two-dimensional image to obtain the third position information of each target object may include inputting at least one two-dimensional image into a preset second neural network model, and outputting the third position information of each target object through the preset second neural network model, but is not limited to this.

[0100] The second neural network model can be configured according to actual conditions, and this embodiment does not impose any specific limitation on this.

[0101] For example, the second neural network model may be Yolov3; or, the second neural network model may be Faster-RCNN, but is not limited thereto.

[0102] The fourth position information of the target object refers to the position information of the target object in the world coordinate system. The fourth position information of the target object can be understood as the spatial coordinates of the target object in the target area.

[0103] The fourth position information of the target object is used to participate in the generation process of the three-dimensional point cloud model.

[0104] The above-mentioned conversion processing of the third position information of each target object to obtain the fourth position information of each target object may include performing an affine transformation on the third position information of each target object to obtain the fourth position information of each target object.

[0105] The above step A101, the above step A102 and the above step A104 may be performed in parallel, that is, the above step A101, the above step A102 and the above step A104 are performed at the same time.

[0106] Alternatively, the above step A101, the above step A102 and the above step A104 may be performed in series, for example, the above step A101 is performed first, then the above step A102 is performed, and finally the above step A104 is performed, but it is not limited thereto.

[0107] The above step A101, the above step A103 and the above step A104 may be performed in parallel, that is, the above step A101, the above step A103 and the above step A104 are performed at the same time.

[0108] Alternatively, the above step A101, the above step A103 and the above step A104 may be performed in series, for example, the above step A101 is performed first, then the above step A103 is performed, and finally the above step A104 is performed, but it is not limited thereto.

[0109] The above-mentioned fusing the depth point cloud data, radar point cloud data, the shape features of each target object and the fourth position information of each target object to obtain the three-dimensional point cloud model of each target object may include fusing the depth point cloud data, radar point cloud data, the shape features of each target object and the fourth position information of each target object through a three-dimensional reconstruction algorithm to generate a three-dimensional point cloud model of each target object.

[0110] In some embodiments, reference Figure 4 In the above step A101, the process of extracting features from at least one two-dimensional image to obtain shape features of each target object may include the following steps A1011-A1013:

[0111] A1011, performing segmentation processing on at least one two-dimensional image by using a Mask-RCNN model to obtain foreground information and instance segmentation masks of each target object;

[0112] A1012, performing edge detection on the foreground information and instance segmentation mask of each target object by using the Canny operator to obtain edge information and contour information of each target object;

[0113] A1013, extracting edge information and contour information of each target object through the Hu feature extraction algorithm, and obtaining edge features and contour features of each target object as shape features of each target object.

[0114] In this embodiment, first, the Mask-RCNN model is used to perform segmentation processing on at least one two-dimensional image, aiming to segment the target object and the background area in the two-dimensional image, thereby obtaining the foreground information and instance segmentation mask of each target object; then, the Canny operator is used to perform edge detection on the foreground information and instance segmentation mask of each target object, so as to obtain the edge information and contour information of each target object; finally, the Hu feature extraction algorithm is used to extract the edge information and contour information of each target object, so as to obtain the edge features and contour features of each target object as the shape features of each target object, so as to capture more object contour details and object edge details, and effectively improve the extraction accuracy of shape features.

[0115] In some implementations, in the above step A103, the process of converting the third position information of each target object to obtain the fourth position information of each target object may include the following step A1031:

[0116] A1031, using a homogeneous coordinate transformation formula to transform the third position information of each target object into the fourth position information of each target object, wherein the fourth position information of the target object satisfies the following formula (1):

[0117] P world =K -1 *P image *Z+T cam (1);

[0118] In formula (1), P world represents the fourth position information of the target object; K represents the internal parameter matrix of the image acquisition module; P image represents the third position information of the target object; Z represents the distance between the target object and the image acquisition module; T cam Represents the coordinate system transformation matrix.

[0119] In this embodiment, the third position information of each target object is converted into the fourth position information of each target object through the homogeneous coordinate transformation formula, thereby converting the position information from two-dimensional space to three-dimensional space, which is conducive to the generation of a three-dimensional point cloud model.

[0120] The distance between the target object and the image acquisition module may be measured by a device such as an ultrasonic distance measuring sensor, but is not limited thereto.

[0121] The above coordinate system transformation matrix is ​​used to represent the translation relationship between the camera coordinate system and the world coordinate system.

[0122] The above-mentioned coordinate system conversion matrix and the above-mentioned internal parameter matrix can be pre-calibrated according to actual conditions, and this embodiment does not make any specific limitation to this.

[0123] In some embodiments, reference Figure 5 In the above step A105, the process of fusing the depth point cloud data, the radar point cloud data, the shape features of each target object and the fourth position information of each target object to obtain the three-dimensional point cloud model of each target object may include the following steps A1051-A1052:

[0124] A1051, integrating the depth point cloud data, the radar point cloud data, the shape characteristics of each target object and the fourth position information of each target object into three-dimensional structure information of the target object;

[0125] A1052, uses an open source point cloud processing library to process the three-dimensional structure information of the target object and generate a three-dimensional point cloud model of the target object.

[0126] In this embodiment, for each target object, first, the depth point cloud data, radar point cloud data, shape features of each target object and the fourth position information of each target object are integrated into the three-dimensional structure information of the target object, and the three-dimensional structure information can represent the position information and shape features of the target object; then, the three-dimensional structure information of the target object is processed by PCL (Point Cloud Library, open source point cloud processing library) to generate a three-dimensional point cloud model of the target object, thereby achieving efficient and accurate generation and processing of the three-dimensional point cloud model.

[0127] In one example, the three-dimensional point cloud model of the target object satisfies the following formula (2):

[0128] P aligned =RP Source +T align (2);

[0129] In formula (2), P alignedrepresents the three-dimensional point cloud model of the target object, that is, the three-dimensional position information and shape characteristics of the target object in the target area; R represents the rotation matrix of the image acquisition module at different viewing angles; P source Represents the spatial position and shape of the depth point cloud data and radar point cloud data in the target area; T align Represents the translation transformation matrix of the depth point cloud data and radar point cloud data in the target area.

[0130] The above rotation matrix and the above translation transformation matrix can be pre-calibrated according to actual conditions, and this embodiment does not make any specific limitation to this.

[0131] In some embodiments, reference Figure 6 In the above step S1022, the process of obtaining the number of target objects and the height information and spacing information of each target object according to the three-dimensional point cloud model of each target object may include the following steps B101-B103:

[0132] B101, performing point cloud clustering processing on the three-dimensional point cloud model of each target object to obtain the number of target objects;

[0133] B102, obtaining height information of the target object according to the 3D point cloud depth set of each target object, wherein the 3D point cloud depth set includes a depth value of each 3D point cloud data in the 3D point cloud model;

[0134] B103, obtaining the distance information of the target object according to the three-dimensional position information of the target object and the three-dimensional position information of the adjacent objects.

[0135] In this embodiment, the number of target objects is obtained by performing point cloud clustering processing on the three-dimensional point cloud models of each target object, and the height information of the target object is obtained through the three-dimensional point cloud depth set of each target object. The three-dimensional point cloud depth set includes the depth value of each three-dimensional point cloud data in the three-dimensional point cloud model, and the distance information of the target object is obtained through the three-dimensional position information of the target object and the three-dimensional position information of adjacent objects. In this way, based on the three-dimensional point cloud models of each target object, the number of target objects and the height information and distance information of each target object are extracted and processed, thereby effectively improving the extraction efficiency and accuracy of stacking characteristic data.

[0136] The above-mentioned point cloud clustering processing is performed on the three-dimensional point cloud model of each target object to obtain the number of target objects, which may include processing the three-dimensional point cloud data in the three-dimensional point cloud model of each target object by a clustering algorithm to obtain the number of target objects. The number of target objects satisfies the following formula (3):

[0137]

[0138] In formula (3), N represents the number of target objects, that is, the total number of target objects piled in the target area; P i represents all 3D point cloud data; k represents the total number of 3D point cloud data; Cluster(P i ) represents the processing process of the clustering algorithm, which takes all 3D point cloud data as the input of the clustering algorithm, clusters all 3D point cloud data through the clustering algorithm, identifies each independent target object, and thus obtains the number of target objects.

[0139] The above clustering algorithm can be set according to actual conditions, and this embodiment does not make any specific limitation to this.

[0140] For example, the clustering algorithm may be a Gaussian Mixture Models (GMM); or, the clustering algorithm may be a Density-Based Spatial Clustering of Applications with Noise (DBSCAN) algorithm, but is not limited thereto.

[0141] The above-mentioned three-dimensional point cloud depth set may include the depth value of each three-dimensional point cloud data in the three-dimensional point cloud model.

[0142] The depth value of the above-mentioned three-dimensional point cloud data can be obtained by aligning the depth point cloud data and the radar point cloud data through an iterative closest point algorithm (IPC) so that the radar point cloud data have corresponding depth values, and the depth point cloud data itself also has corresponding depth values. In this way, after generating a three-dimensional point cloud model, each three-dimensional point cloud data in the three-dimensional point cloud model also has a corresponding depth value, but is not limited to this.

[0143] The above-mentioned obtaining the height information of the target object based on the three-dimensional point cloud depth set of each target object can include, for each target object, selecting the three-dimensional point cloud data with the largest depth value from the three-dimensional point cloud depth set as the target point cloud data, and using the depth value of the target point cloud data as the height information of the target object.

[0144] Alternatively, obtaining the height information of the target object according to the three-dimensional point cloud depth set of each target object may include, for each target object, selecting three-dimensional point cloud data with the largest depth value from the three-dimensional point cloud depth set as the first point cloud data, and selecting three-dimensional point cloud data with the smallest depth value as the second point cloud data, and calculating the difference between the depth value of the first point cloud data and the depth value of the second point cloud data as the height information of the target object, wherein the height information of the target object satisfies the following formula (4):

[0145] H=max(Zpoints )-min(Z points ) (4);

[0146] In formula (4), H represents the height information of the target object; Z points Represents the 3D point cloud depth set of the target object; max(Z points ) represents the depth value of the first point cloud data; min(Z points ) represents the depth value of the second point cloud data.

[0147] The above-mentioned obtaining the distance information of the target object based on the three-dimensional position information of the target object and the three-dimensional position information of the adjacent objects may include obtaining the distance information of the target object based on the three-dimensional position information of the target object and the three-dimensional position information of the adjacent objects in combination with a machine learning method.

[0148] Alternatively, obtaining the distance information of the target object according to the three-dimensional position information of the target object and the three-dimensional position information of the adjacent objects may include calculating the Euclidean distance between the target object and the adjacent objects as the distance information of the target object according to the three-dimensional position information of the target object and the three-dimensional position information of the adjacent objects, and the distance information of the target object satisfies the following formula (5):

[0149]

[0150] In formula (5), d ij represents the distance information of the i-th target object, that is, the Euclidean distance between the i-th target object and the j-th target object. The i-th target object and the j-th target object are adjacent objects to each other; (x i ,y i , z i ) represents the three-dimensional position information of the i-th target object. Optionally, the three-dimensional position information of the i-th target object refers to the spatial coordinates of the i-th target object in the target area; (x j ,y j , z j ) represents the three-dimensional coordinate information of the j-th target object. Optionally, the three-dimensional coordinate information of the j-th target object in the target area refers to the spatial coordinates of the j-th target object.

[0151] In some embodiments, in the above step S1023, the process of performing object detection on at least one two-dimensional image to obtain the second position information and category information of each target object may include the following step C101:

[0152] C101, using a pre-trained object detection model, perform object detection on at least one two-dimensional image to obtain second position information and category information of each target object.

[0153] In this embodiment, at least one two-dimensional image is input into a pre-trained object detection model, and object detection is performed through the pre-trained object detection model to obtain second position information and category information of each target object. This can efficiently and accurately locate the position and category of the target object in the target area, so that the three-dimensional position information can be combined in the subsequent steps to generate the first position information of the target object.

[0154] The object detection model may be a neural network model trained by using a plurality of preset two-dimensional image samples and an object label corresponding to each preset two-dimensional image sample. The object label may include the position information of the object in the image sample and the category information of the object.

[0155] The above object detection model can be set according to actual conditions, and this embodiment does not make any specific limitation to this.

[0156] For example, the object detection model may be EfficientDet; or, the object detection model may be a convolutional neural network model.

[0157] In one example, the object detection model is a pre-trained EfficientDet. At least one two-dimensional image is input into EfficientDet, and target detection is performed by EfficientDet. The target detection process satisfies the following formula (6):

[0158] BBox pos =W pos *F+b pos , BBox size =W size *F+b size (6);

[0159] In formula (6), the target object contained in the two-dimensional image is defined as the detection object, and BBox pos represents the object position of the two-dimensional image, that is, the position information of each detected object in the two-dimensional image; W pos represents the position weight, b pos represents position bias; F represents feature map; BBox size Indicates the object size of the two-dimensional image, that is, the size information of each detected object; W size represents the size weight, b size It should be understood that the position weight, position bias, size weight and size bias can all be set according to actual conditions.

[0160] At the same time, bounding box regression and classification are performed through EfficientDet, and the bounding box regression and classification satisfy the following formula (7):

[0161] BoundingBox=[x,y,w,h],ClassScores=Softmax(W cls *F+b cls ) (7);

[0162] In formula (7), BoundingBox represents the bounding box coordinates, which are used to characterize the position and size of each detected object in the two-dimensional image. The bounding box coordinates are mainly used to evaluate the performance of EfficientDet and determine the object position and object size of each two-dimensional image; (x, y) represents the center coordinates of the bounding box, w represents the width of the bounding box, and h represents the height of the bounding box; ClassScores represents the object category probability of the two-dimensional image. The object category probability may include the probability that each detected object belongs to each category. The object category probability is mainly used to determine the category information of each detected object; W cls represents the classification weight, b cls It should be understood that both the classification weight and the classification bias can be set according to actual conditions.

[0163] After obtaining the object positions of each two-dimensional image, for each target object, the second position information of the target object is determined according to the object position of at least one two-dimensional image containing the target object. For example, for each target object, from at least one two-dimensional image containing the target object, a two-dimensional image containing the target object is arbitrarily selected as the target image, and the object position of the target image is determined as the second position information of the target object. For another example, for each target object, the average of the object positions of at least one two-dimensional image containing the target object is obtained as the second position information of the target object, but it is not limited thereto.

[0164] And, after obtaining the object category probability of each two-dimensional image, for each target object, the category information of the target object is determined according to the object category probability of at least one two-dimensional image containing the target object. For example, for each target object, from at least one two-dimensional image containing the target object, a two-dimensional image containing the target object is arbitrarily selected as the target image, the maximum value of the probability that the target object belongs to each category is determined as the target probability, and the category corresponding to the target probability is used as the category information of the target object. For another example, for each target object, a sub-bounding box is drawn according to the shape feature of the target object, and the IoU value of each two-dimensional image is obtained according to the sub-bounding box and the bounding box of each two-dimensional image, and then the two-dimensional image with the highest IoU value is used as the target image, the maximum value of the probability that the target object belongs to each category is determined as the target probability, and the category corresponding to the target probability is used as the category information of the target object, wherein the IoU value of the two-dimensional image satisfies the following formula (8):

[0165]

[0166] In formula (8), IoU represents the IoU value of the two-dimensional image, which indicates the degree of overlap between the sub-bounding box and the bounding box; R 1 ∩B 2 Represents the sub-bounding box B 1 and bounding box B 2 The intersection area of ​​B 1 ∪B 2 Represents the sub-bounding box B 1 and bounding box B 2 The union area of ​​; Area() represents the area function.

[0167] In some embodiments, reference Figure 7 In the above step S1024, the process of obtaining the first position information of each target object according to the second position information and the three-dimensional position information of each target object may include the following step D101:

[0168] D101 , for each target object, weight the second position information and the three-dimensional position information to obtain the first position information of the target object.

[0169] In this embodiment, for each target object, the second position information obtained by the object detection technology and the three-dimensional position information obtained by the three-dimensional modeling technology are used to determine the final position information of the target object, that is, the first position information. In this way, the accuracy of the position information of the target object can be effectively improved.

[0170] The first position information of the target object satisfies the following formula (9):

[0171] P final =αP detection +βP feature (9);

[0172] In formula (9), P final Represents the first position information of the target object; P detection represents the second position information obtained by the object detection technology; P feature represents the three-dimensional position information obtained by the three-dimensional modeling technology; α is the weight of the second position information; β is the weight of the three-dimensional position information.

[0173] The weight of the second position information and the weight of the three-dimensional position information can be set according to actual conditions, and this embodiment does not specifically limit this.

[0174] In some embodiments, reference Figure 8In the above step S103, according to the number of target objects and the first position information, height information, category information and spacing information of each target object, the implementation process of detecting abnormal stacking of objects in the target area may include at least one of the following steps S1031-S1034:

[0175] S1031, if the detected number of target objects is greater than a first preset threshold, determining that a preset stacking number abnormality exists in the target area;

[0176] S1032, if at least one target object whose first position information is outside a preset range is detected, determining that a preset stacking position abnormality exists in the target area, where the preset range is determined by category information of the target object;

[0177] S1033, if at least one target object whose height information is greater than a second preset threshold is detected, it is determined that a preset stacking height abnormality exists in the target area;

[0178] S1034: If at least one target object whose spacing information is less than a third preset threshold is detected, it is determined that a preset stacking spacing anomaly exists in the target area.

[0179] In this implementation, object stacking anomaly detection is implemented from four aspects: excessive stacking, misplaced stacking, excessively high stacking, and overly dense stacking.

[0180] Specifically, for the detection of excess stacking, it is determined whether the number of target objects is greater than a first preset threshold. If so, it means that the target objects stacked in the target area are stacked in excess, and it is determined that there is a preset stacking quantity abnormality in the target area.

[0181] For the detection of stacking misalignment, the preset range of each target object is first determined through the category information of each target object, and then it is determined whether the first position information of each target object is outside the preset range. If there is at least one target object whose first position information is outside the preset range, it means that at least one target object is stacked misaligned in the target area. At this time, it is determined that there is a preset stacking position abnormality in the target area.

[0182] For the detection of excessive stacking, it is determined whether the height information of each target object is greater than a second preset threshold. If there is at least one target object whose height information is greater than the second preset threshold, it means that at least one target object in the target area is stacked excessively high. At this time, it is determined that there is a preset stacking height abnormality in the target area.

[0183] For the detection of over-crowded stacking, it is determined whether the spacing information of each target object is less than a third preset threshold. If there is at least one target object whose spacing information is less than the third preset threshold, it means that at least two adjacent target objects are stacked over-crowded in the target area. At this time, it is determined that there is a preset stacking spacing anomaly in the target area.

[0184] The above-mentioned stacking quantity abnormality is used to indicate that the total number of target objects stacked in the target area exceeds a first preset threshold, that is, the stacking is excessive.

[0185] The stacking height anomaly is used to indicate that there is at least one target object in the target area that is not stacked at the designated location, that is, the stacking is misplaced.

[0186] The stacking height anomaly is used to indicate that at least one target object in the target area is stacked too high, that is, the stacking is too high.

[0187] The above-mentioned stacking spacing anomaly is used to indicate that the distance between at least two adjacent target objects in the target area is too small, that is, the stacking is too dense.

[0188] The preset ranges correspond to the target objects one by one, that is, each target object is provided with a corresponding preset range.

[0189] The above-mentioned preset range, the above-mentioned first preset threshold, the above-mentioned second preset threshold and the above-mentioned third preset threshold can all be set according to actual conditions, and this embodiment does not make any specific limitation on this.

[0190] For example, the first preset threshold is 5, the second preset threshold is 2 meters, and the third preset threshold is 0.5 meters, but it is not limited thereto.

[0191] The above steps S1031 to S1034 may be performed in parallel, that is, steps S1031 to S1034 are performed simultaneously.

[0192] Alternatively, the above steps S1031 to S1034 may be performed serially, for example, by performing steps S1031 to S1034 in sequence, but the present invention is not limited thereto.

[0193] To facilitate the understanding of the above-mentioned object stacking abnormality detection method of the present application, the implementation of the above-mentioned object stacking abnormality detection method of the present application will be illustrated below with examples.

[0194] The application scenario of this example is an industrial workshop, which is equipped with a storage area. Currently, five target objects, box a, box b, bucket c, tool d and equipment e, are stacked in the storage area. The requirements for object stacking in this storage area include: the stacking height of objects cannot exceed 2 meters, the total number of objects cannot exceed 5, and the distance between two adjacent objects cannot exceed 0.5 meters. Multiple three-primary color channel cameras, depth cameras and laser radars are set in this storage area. The image acquisition angle of each three-primary color channel camera is different, and the depth acquisition range of the depth camera is 1.5 meters to 2.5 meters.

[0195] Based on this, the process of object stacking anomaly detection in this example is specifically shown in the following steps S201-S204.

[0196] S201, data collection:

[0197] The target area is imaged by multiple three-primary color channel cameras to obtain at least one two-dimensional image, the target area is scanned by at least one laser radar to obtain radar point cloud data, and the target area is scanned by at least one depth camera to obtain depth data.

[0198] In this example, the depth data of box a is 1.8 meters, the depth data of box b is 1.9 meters, the depth data of bucket c is 2 meters, the depth data of tool d is 1.7 meters, and the depth data of device e is 2.2 meters. The radar point cloud data of box a includes 5,000 points, the radar point cloud data of box b includes 5,100 points, the radar point cloud data of bucket c includes 7,000 points, the radar point cloud data of tool d includes 3,000 points, and the radar point cloud data of device e includes 10,000 points.

[0199] S202, constructing a three-dimensional point cloud model of each target object:

[0200] Firstly, the Mask-RCNN model is used to perform segmentation processing on at least one two-dimensional image, aiming to segment the target object and the background area in the two-dimensional image, so as to obtain the foreground information and instance segmentation mask of each target object; then, the Canny operator is used to perform edge detection on the foreground information and instance segmentation mask of each target object, so as to obtain the edge information and contour information of each target object; finally, the Hu feature extraction algorithm is used to extract the edge information and contour information of each target object, so as to obtain the edge features and contour features of each target object as the shape features of each target object.

[0201] At the same time, at least one two-dimensional image is input into a preset second neural network model, and the third position information of each target object is output through the preset second neural network model. Then, the third position information of each target object is converted into the fourth position information of each target object using the homogeneous coordinate transformation formula shown in the above formula (1).

[0202] In this example, the fourth position information of box a is coordinates (1.0, 2.0, 0.5), the fourth position information of box b is coordinates (2.0, 1.5, 0.5), the fourth position information of bucket c is coordinates (1.5, 3.0, 0.7), the fourth position information of tool d is coordinates (2.5, 2.5, 0.3), and the fourth position information of device e is coordinates (3.0, 1.0, 1.0).

[0203] At the same time, the depth data of each target object is converted into depth point cloud data, and the ICP algorithm is used to perform point cloud registration on the depth point cloud data and the radar point cloud data, so that the radar point cloud data has corresponding depth values, and the depth point cloud data itself also has corresponding depth values.

[0204] Afterwards, the depth point cloud data, radar point cloud data, shape features of each target object and the fourth position information of each target object are integrated into the three-dimensional structure information of each target object. The three-dimensional structure information can represent the position information and shape features of the target object. The three-dimensional structure information of each target object is processed by PCL to generate a three-dimensional point cloud model of each target object. The three-dimensional point cloud model is as shown in the above formula (2). The three-dimensional point cloud model of the target object includes a plurality of three-dimensional point cloud data, and the plurality of three-dimensional point cloud data jointly represent the shape features and three-dimensional position information of the target object.

[0205] S203, extracting the stacking characteristic data of each target object:

[0206] The three-dimensional point cloud data in the three-dimensional point cloud model of each target object is processed by a clustering algorithm to obtain the number of target objects, as shown in the above formula (3).

[0207] In this example, clustering processing can determine that the number of target objects is 5.

[0208] At the same time, for each target object, the following is done: the three-dimensional point cloud data with the largest depth value is selected from the three-dimensional point cloud depth set as the first point cloud data, and the three-dimensional point cloud data with the smallest depth value is selected as the second point cloud data, and the difference between the depth value of the first point cloud data and the depth value of the second point cloud data is calculated as the height information of the target object, as described in the above formula (4), and based on the three-dimensional position information of the target object and the three-dimensional position information of the adjacent objects, the Euclidean distance between the target object and the adjacent objects is calculated as the spacing information of the target object, as shown in the above formula (5), wherein the three-dimensional position information refers to the spatial coordinates of the object in the target area.

[0209] In this example, by analyzing the 3D point cloud data, the following data can be obtained: the maximum depth value of box a is 0.7 meters and the minimum depth value is 0.2 meters, so the height information of box a is 0.5 meters; the maximum depth value of box b is 0.8 meters and the minimum depth value is 0.3 meters, so the height information of box b is 0.5 meters; the maximum depth value of bucket c is 1.2 meters and the minimum depth value is 0.5 meters, so the height information of bucket c is 0.7 meters; the maximum depth value of tool d is 0.5 meters and the minimum depth value is 0.2 meters, so the height information of tool d is 0.3 meters; the maximum depth value of device e is 2.2 meters and the minimum depth value is 1.2 meters, so the height information of device e is 1 meter. Box a and box b are adjacent, and the distance between box a and box b is 1.12 meters; bucket c and tool d are adjacent, and the distance between bucket c and tool d is 1.19 meters; device e and box a are adjacent, and the distance between device e and box a is 2.29 meters.

[0210] At the same time, at least one two-dimensional image is input into the pre-trained EfficientDet to perform object detection through EfficientDet. The object detection process is shown in the above formula (6). At the same time, bounding box regression and classification are performed through EfficientDet, as shown in the above formula (7), so as to obtain the category information and second position information of each target object.

[0211] In this example, through object detection, we can get the following data: the object category probability of box a is 0.95, the bounding box coordinates are (0.9, 1.9, 1.1, 2.1), and the size is 0.5m×0.5m×0.5m; the object category probability of box b is 0.90, the bounding box coordinates are (1.9, 1.4, 2.1, 1.6), and the size is 0.5m×0.5m×0.5m; the object category probability of bucket c is 0.92, and the bounding box The coordinates are (1.4, 2.9, 1.6, 3.1) and the size is 0.4m×0.4m×0.7m; the object category probability of tool d is 0.88, the bounding box coordinates are (2.4, 2.4, 2.6, 2.6), and the size is 0.3m×0.3m×0.3m; the object category probability of device e is 0.93, the bounding box coordinates are (2.9, 0.9, 3.1, 1.1), and the size is 1.0m×0.5m×1.0m.

[0212] Through bounding box regression and classification, we can obtain the following data: the second position information of box a is (1.0, 2.0, 0.2, 0.2), and the category is "box"; the second position information of box b is (2.0, 1.5, 0.2, 0.2), and the category is "box"; the second position information of bucket c is (1.5, 3.0, 0.2, 0.2), and the category is "bucket"; the second position information of tool d is (2.5, 2.5, 0.2, 0.2), and the category is "tool"; the second position information of equipment e is (3.0, 1.0, 0.2, 0.2), and the category is "equipment".

[0213] Subsequently, the three-dimensional position information and the second position information of each target object are weighted, as shown in the above formula (9), so as to obtain the first position information of each target object, that is, the position information of each target object in the target area.

[0214] S204, object stacking anomaly detection:

[0215] The preset range of each target object is determined by the category information of each target object, and the first position information of each target object is compared to see whether it is within the preset range. If so, it means that there is no abnormal stacking position in the target area, otherwise it means that there is an abnormal stacking position in the target area, and an alarm signal is issued at this time. In this example, the first position information of each target object is within the corresponding preset range, so there is no abnormal stacking position in the target area.

[0216] Compare the number of target objects with the first preset threshold. If the number of target objects is not greater than the first preset threshold, it means that there is no abnormal number of stacking in the target area. Otherwise, it means that there is an abnormal number of stacking in the target area, and an alarm signal is issued. In this example, the first preset threshold is 5, and the total number of target objects is equal to 5, so there is no abnormal number of stacking in the target area.

[0217] Compare the height information of each target object with the second preset threshold. If the height information of all target objects is less than or equal to the second preset threshold, it means that there is no abnormal stacking height in the target area. Otherwise, it means that there is an abnormal stacking height in the target area, and an alarm signal is issued. In this example, the second preset threshold is 5 meters, the height information of box a is 0.5 meters, the height information of box b is 0.5 meters, the height information of bucket c is 0.7 meters, the height information of tool d is 0.3 meters, and the height information of equipment e is 1 meter. The height information of all target objects is less than 5 meters, so there is no abnormal stacking height in the target area.

[0218] Compare the spacing information of each target object with the third preset threshold. If the spacing information of all target objects is greater than or equal to the third preset threshold, it means that there is no abnormal stacking spacing in the target area. Otherwise, it means that there is an abnormal stacking spacing in the target area, and an alarm signal is issued. In this example, the third preset threshold is 0.5 meters, box a and box b are adjacent, the spacing between box a and box b is 1.12 meters, and the spacing between box a and box b is greater than 0.5 meters; bucket c and tool d are adjacent, the spacing between bucket c and tool d is 1.19 meters, and the spacing between bucket c and tool d is greater than 0.5 meters; equipment e and box a are adjacent, the spacing between equipment e and box a is 2.29 meters, and the spacing between equipment e and box a is greater than 0.5 meters, so there is no abnormal stacking spacing in the target area.

[0219] In summary, it can be seen that the embodiments of the present application, while fully considering multimodal data such as two-dimensional images, point cloud data and depth data, extract the stacking characteristic data of each target object, such as the number of target objects and the first position information, height information, category information and spacing information of each target object, and on the basis of the stacking characteristic data of each target object, realize real-time monitoring of object stacking and automatic detection of abnormal object stacking. In this way, the accuracy and efficiency of abnormal object stacking detection can be effectively improved, and the safety risks caused by abnormal object stacking can be reduced.

[0220] Secondly, refer to Fig. 9 , the embodiment of the present application also provides an object stacking abnormality detection device, including:

[0221] The acquisition module 301 is used to acquire depth data, radar point cloud data and at least one two-dimensional image of a target area; a plurality of target objects are stacked in the target area;

[0222] A first processing module 302 is used to obtain the number of target objects and first position information, height information, category information and spacing information of each target object according to the depth data, the radar point cloud data and the at least one two-dimensional image, wherein the spacing information of the target object is the spacing between the target object and an adjacent object, the adjacent object is adjacent to the target object, and the first position information of the target object is the position information of the target object in the target area;

[0223] The second processing module 303 is used to perform object stacking abnormality detection on the target area according to the number of target objects and the first position information, height information, category information and spacing information of each target object.

[0224] In some implementations, the acquisition module 301 may include:

[0225] at least one camera, disposed within the target area, for capturing at least one two-dimensional image of the target area;

[0226] At least one laser radar, arranged in the target area, for collecting radar point cloud data of the target area;

[0227] At least one depth camera is arranged in the target area and is used to collect depth data of the target area.

[0228] In some implementations, the second processing module 303 may include:

[0229] A position analysis unit, used to detect whether there is any abnormality in the preset stacking position in the target area;

[0230] The state analysis unit is used to detect whether there is a preset stacking quantity abnormality, a preset stacking height abnormality, and a preset stacking distance abnormality in the target area.

[0231] In some embodiments, the above device may further include:

[0232] An alarm triggering unit, configured to generate an alarm signal if at least one of an abnormal stacking position, an abnormal stacking quantity, an abnormal stacking height, or an abnormal stacking spacing is detected in the target area;

[0233] An alarm device, used to output an alarm signal in the form of sound or light or by communication transmission;

[0234] In some embodiments, the above device may further include:

[0235] a parameter setting unit, used to adjust a first preset threshold, a second preset threshold, a third preset threshold and a preset range of a target object;

[0236] A real-time monitoring unit, used to display the monitoring screen of the target area and the detection results of abnormal stacking of objects in the target area;

[0237] The alarm display unit is used to display the alarm information of abnormal object stacking, the alarm information includes current alarm information and historical alarm information, the alarm information may include the type of abnormal object stacking, abnormal time and abnormal target area, and record the alarm log.

[0238] The contents of the above method embodiments are all applicable to the present device embodiments. The functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0239] In addition, refer to Fig.10 , an embodiment of the present application provides an electronic device, which may include:

[0240] at least one processor 401;

[0241] At least one memory 402, for storing at least one program;

[0242] When at least one program is executed by at least one processor 401, at least one processor 401 implements the above-mentioned object stacking abnormality detection method.

[0243] The memory 402, as a non-transient network system, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 402 may optionally include a memory 402 remotely arranged relative to the processor 401, and these remote memories 402 may be connected to the processor 401 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0244] The memory 402 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 402 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 402, and the processor 401 calls and executes the methods of the embodiments of this application.

[0245] The processor 401 can be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0246] In some implementations, the electronic device may further include:

[0247] Input / output interface, used to realize information input and output;

[0248] Communication interface, used to realize communication interaction between this device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0249] A bus that transmits information between various components of the device (e.g., processor 401, memory 402, input / output interface, and communication interface);

[0250] The processor 401 , the memory 402 , the input / output interface and the communication interface can be connected to each other in communication within the device via a bus.

[0251] The contents of the above method embodiments are all applicable to the present device embodiments. The functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0252] Finally, an embodiment of the present application provides a computer-readable storage medium, which stores a program executable by a processor. When the program executable by the processor is executed by the processor, it is used to implement the above-mentioned object stacking abnormality detection method.

[0253] The contents of the above method embodiments are all applicable to the present storage medium embodiments. The functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0254] In some optional embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the application is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Optional embodiments are expected, wherein the order of various operations is changed and the sub-operation described as a part of a larger operation is performed independently.

[0255] In addition, although the present application is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features can be integrated into a single physical device and / or software module, or one or more functions and / or features can be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present application. More specifically, in view of the properties, functions, and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional techniques of the engineer. Therefore, those skilled in the art can implement the present application set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present application, which is determined by the full scope of the attached claims and their equivalents.

[0256] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several programs to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (R AM), a magnetic disk or an optical disk.

[0257] The logic and / or steps represented in the flowchart or otherwise described herein, for example, may be considered as an ordered list of executable programs for implementing the logical functions, and may be embodied in any computer-readable medium for use by a program execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch and execute a program from a program execution system, device or apparatus), or in conjunction with such program execution systems, devices or apparatuses. For purposes of this specification, a "computer-readable medium" may be any device that can contain, store, communicate, propagate or transmit a program for use by a program execution system, device or apparatus, or in conjunction with such program execution systems, devices or apparatuses.

[0258] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0259] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable program execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0260] In the above description of this specification, the description with reference to the terms "one embodiment / example", "another embodiment / example" or "certain embodiments / examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0261] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

[0262] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the described embodiments. Technical personnel familiar with the field may make various equivalent modifications or substitutions without violating the spirit of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.

Claims

1. A method for detecting abnormal stacking of objects, characterized in that: The following steps are involved: Acquire depth data, radar point cloud data and at least one two-dimensional image of a target area; the target area has a plurality of target objects stacked therein; According to the depth data, the radar point cloud data and at least one of the two-dimensional images, obtain the number of target objects and first position information, height information, category information and spacing information of each of the target objects, wherein the spacing information of the target objects is the spacing between the target object and an adjacent object, the adjacent object is adjacent to the target object, and the first position information of the target object is the position information of the target object in the target area; According to the number of target objects and the first position information, height information, category information and spacing information of each target object, object stacking abnormality detection is performed on the target area.

2. The object stacking abnormality detection method according to claim 1, characterized in that: The obtaining, according to the depth data, the radar point cloud data and at least one of the two-dimensional images, the number of target objects and the first position information, height information, category information and spacing information of each of the target objects comprises: The depth data, the radar point cloud data and at least one of the two-dimensional images are fused to obtain a three-dimensional point cloud model of each of the target objects; the three-dimensional point cloud model of the target object includes a plurality of three-dimensional point cloud data, and the three-dimensional point cloud model of the target object is used to characterize the three-dimensional position information and shape characteristics of the target object; According to the three-dimensional point cloud model of each target object, the number of the target objects and the height information and the distance information of each target object are obtained; Performing object detection on at least one of the two-dimensional images to obtain second position information and category information of each of the target objects; the second position information of the target object is position information of the target object in the two-dimensional image; The first position information of each target object is obtained according to the second position information and the three-dimensional position information of each target object.

3. The object stacking abnormality detection method according to claim 2, characterized in that: The fusing the depth data, the radar point cloud data and at least one of the two-dimensional images to obtain a three-dimensional point cloud model of each of the target objects comprises: Performing feature extraction on at least one of the two-dimensional images to obtain shape features of each of the target objects, wherein the shape features include at least one of edge features or contour features; Performing target recognition processing on at least one of the two-dimensional images to obtain third position information of each of the target objects, wherein the third position information of the target object includes pixel coordinates of the target object in each of the two-dimensional images; The third position information of each target object is converted to obtain fourth position information of each target object; the fourth position information of the target object is the position information of the target object in the world coordinate system; Performing point cloud conversion processing on the depth data to obtain depth point cloud data; The depth point cloud data, the radar point cloud data, the shape features of each of the target objects and the fourth position information of each of the target objects are fused to obtain a three-dimensional point cloud model of each of the target objects.

4. The object stacking abnormality detection method according to claim 2, characterized in that: The step of obtaining the number of target objects and the height information and the spacing information of each target object according to the three-dimensional point cloud model of each target object includes: Performing point cloud clustering processing on the three-dimensional point cloud model of each target object to obtain the number of target objects; Obtaining height information of the target object according to a three-dimensional point cloud depth set of each target object, wherein the three-dimensional point cloud depth set includes a depth value of each three-dimensional point cloud data in the three-dimensional point cloud model; The distance information of the target object is obtained according to the three-dimensional position information of the target object and the three-dimensional position information of the adjacent objects.

5. The object stacking abnormality detection method according to claim 2, characterized in that: The performing object detection on at least one of the two-dimensional images to obtain second position information and category information of each of the target objects includes: Using a pre-trained object detection model, object detection is performed on at least one of the two-dimensional images to obtain second position information and category information of each of the target objects.

6. The object stacking abnormality detection method according to claim 2, characterized in that: The obtaining the first position information of each target object according to the second position information and the three-dimensional position information of each target object comprises: For each of the target objects, the second position information and the three-dimensional position information are weighted to obtain the first position information of the target object.

7. The object stacking abnormality detection method according to claim 1, characterized in that: The detecting of abnormal stacking of objects in the target area according to the number of target objects and the first position information, height information, category information and spacing information of each target object includes at least one of the following: If it is detected that the number of target objects is greater than a first preset threshold, it is determined that there is a preset stacking quantity abnormality in the target area; If at least one target object whose first position information is outside a preset range is detected, it is determined that a preset stacking position abnormality exists in the target area, and the preset range is determined by the category information of the target object; If at least one target object whose height information is greater than a second preset threshold is detected, it is determined that a preset stacking height abnormality exists in the target area; If at least one target object whose spacing information is less than a third preset threshold is detected, it is determined that a preset stacking spacing anomaly exists in the target area.

8. An object stacking abnormality detection device, characterized in that: include: An acquisition module, used to acquire depth data, radar point cloud data and at least one two-dimensional image of a target area; The target area has a plurality of target objects stacked therein; a first processing module, configured to obtain the number of target objects and first position information, height information, category information and spacing information of each target object according to the depth data, the radar point cloud data and at least one of the two-dimensional images, wherein the spacing information of the target object is the spacing between the target object and an adjacent object, the adjacent object is adjacent to the target object, and the first position information of the target object is the position information of the target object in the target area; The second processing module is used to perform object stacking abnormality detection on the target area according to the number of the target objects and the first position information, height information, category information and spacing information of each of the target objects.

9. An electronic device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the object stacking abnormality detection method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a program executable by a processor, characterized in that: The program executable by the processor is used to implement the object stacking abnormality detection method according to any one of claims 1 to 7 when executed by the processor.

Images