Material shape detection method and device based on multiple cameras and electronic equipment

Abstract

The invention provides a multi-camera-based material shape detection method and device and electronic equipment, and the method comprises the steps: carrying out the image collection of a target material at the same time through a plurality of cameras at different shooting angles in response to an induction triggering operation of the target material, and obtaining an original image of the target material correspondingly collected by each camera; performing target detection on each original image through a preset target detection model to obtain a detection frame of the target material corresponding to each camera; performing image segmentation on an area, corresponding to the corresponding detection frame, on each original image to obtain a shape object, corresponding to each camera, of the target material; respectively comparing each shape object with a preset standard shape object to determine a second classification confidence coefficient of each shape object; and determining a shape detection result of the target material based on the first classification confidence of each detection frame and the second classification confidence of each shape object. According to the invention, the material classification efficiency and precision can be improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method, device and electronic device for detecting the shape of materials based on multiple cameras. Background Art

[0002] In the traditional material sorting process, foreign objects or materials with poor quality are sorted out by manual or semi-automatic equipment. Limited by the limitations of the human eye in observing materials, it is impossible to meet the requirements for sorting efficiency and quality. Therefore, the introduction of vision detection technology to realize the positioning and classification of materials through image recognition can improve the sorting efficiency and quality.

[0003] In the prior art, there are mainly the following two purposes for material classification through vision detection: one is to select materials, that is, to quickly identify defective products in materials and eliminate the defective products; the other is to classify and identify multiple materials, and then sort the multiple materials. When the prior art classifies materials through vision detection, it is necessary for a camera to collect material pictures in real time. As the types of materials increase and the characteristics of materials such as shape and color become diverse and irregular, and the feeding positions and feeding angles of materials on the production line are not fixed, the material pictures collected by a single camera cannot completely describe the true characteristics of the materials, resulting in a very low accuracy of material classification judgment. The material pictures collected by multiple cameras from different angles are difficult to ensure the efficiency and accuracy of material classification due to the large differences in material characteristics. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a method, device and electronic device for detecting the shape of materials based on multiple cameras, so as to improve the efficiency and accuracy of material classification, thereby alleviating the above problems existing in the related art.

[0005] In a first aspect, an embodiment of the present invention provides a method for detecting the shape of materials based on multiple cameras. The method includes: in response to an induction trigger operation of a target material, simultaneously collecting images of the target material through multiple cameras with different shooting angles to obtain the original images of the target material collected by each camera; wherein, the cameras with different shooting angles are fixedly installed at different target positions corresponding to the target material; performing target detection on each original image through a preset target detection model to obtain detection frames of the target material corresponding to each camera; wherein, each detection frame has a corresponding first classification confidence; performing image segmentation on the region corresponding to the corresponding detection frame on each original image to obtain shape objects of the target material corresponding to each camera; respectively comparing each shape object with a preset standard shape object to determine the second classification confidence of each shape object; and determining the shape detection result of the target material based on the first classification confidence of each detection frame and the second classification confidence of each shape object.

[0006] In a second aspect, an embodiment of the present invention further provides a multi-camera based material shape detection device, the device comprising: an image acquisition module, configured to, in response to an induction trigger operation of a target material, simultaneously acquire images of the target material through cameras at multiple different shooting angles, so as to obtain original images of the target material acquired by each camera; wherein, the cameras at different shooting angles are fixedly installed at different target positions corresponding to the target material; a target detection module, configured to perform target detection on each original image through a preset target detection model, so as to obtain detection frames of the target material corresponding to each camera; wherein, each detection frame has a corresponding first classification confidence level; an image segmentation module, configured to perform image segmentation on the region corresponding to the corresponding detection frame on each original image, so as to obtain shape objects of the target material corresponding to each camera; a first determination module, configured to respectively compare each shape object with a preset standard shape object to determine a second classification confidence level of each shape object; a second determination module, configured to determine a shape detection result of the target material based on the first classification confidence levels of the respective detection frames and the second classification confidence levels of the respective shape objects.

[0007] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising a processor and a memory, the memory storing computer-executable instructions capable of being executed by the processor, and the processor executing the computer-executable instructions to implement the multi-camera based material shape detection method described in the first aspect above.

[0008] The multi-camera based material shape detection method, device and electronic device provided by the embodiments of the present invention first trigger multiple cameras at different shooting angles to simultaneously acquire original images of the target material through the induction trigger operation of the target material, then perform target detection on each original image through a preset target detection model to obtain detection frames of the target material corresponding to each camera with the first classification confidence level, then perform image segmentation on the region corresponding to the corresponding detection frame on each original image to obtain shape objects of the target material corresponding to each camera, and respectively compare each shape object with a preset standard shape object to determine the second classification confidence level of each shape object, and finally determine the shape detection result of the target material based on the first classification confidence levels of the respective detection frames and the second classification confidence levels of the respective shape objects. By adopting the above technology, material images acquired by multiple cameras can be used for target detection to obtain material detection frames and their classification confidence levels, and the material detection frames can be used to obtain material shapes and their classification confidence levels through image segmentation, and then the shape detection result of the material can be obtained by combining the classification confidence levels of the material detection frames and the material shapes respectively, thereby improving the efficiency and accuracy of material classification.

[0009] Other features and advantages of the present invention will be set forth in the following description, and in part will be obvious from the description, or may be learned by practice of the present invention. The objectives and other advantages of the present invention are realized and attained by the structure particularly pointed out in the specification, claims and drawings.

[0010] To make the above objectives, features and advantages of the present invention more obvious and understandable, the following specific preferred embodiments are given, in conjunction with the accompanying drawings, and are described in detail as follows. Description of the Drawings

[0011] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for the description of the specific embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0012] Figure 1 It is a schematic flowchart of a method for detecting the shape of materials based on multiple cameras in an embodiment of the present invention;

[0013] Figure 2 It is an example flowchart of a method for detecting the shape of materials based on multiple cameras in an embodiment of the present invention;

[0014] Figure 3 It is a schematic structural diagram of a device for detecting the shape of materials based on multiple cameras in an embodiment of the present invention;

[0015] Figure 4 It is a schematic structural diagram of an electronic device in an embodiment of the present invention. Detailed Embodiments

[0016] To make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions of the present invention in conjunction with the embodiments. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0017] At present, there are mainly the following two purposes for material classification through visual inspection in the prior art: one is to select materials, that is, to quickly identify defective products in the materials and remove the defective products; the other is to classify and identify multiple materials, and then sort the multiple materials. When the prior art performs material classification through visual inspection, it is necessary for the camera to collect material pictures in real time. With the increase in the types of materials and the diversification and irregularity of the characteristics such as the shape and color of the materials, and the fact that the feeding position and feeding angle of the materials on the production line are not fixed, the material pictures collected by a single camera have a very low accuracy in material classification judgment because they cannot completely describe the true characteristics of the materials. The material pictures collected by multiple cameras from different angles are difficult to ensure the efficiency and accuracy of material classification because the material characteristics vary greatly.

[0018] Based on this, an object shape detection method, device and electronic device based on multiple cameras provided by the embodiments of the present invention can improve the efficiency and accuracy of material classification, thereby alleviating the above problems existing in the related art.

[0019] For the convenience of understanding this embodiment, first, a method for detecting the shape of an object based on multiple cameras disclosed in the embodiments of the present invention will be introduced in detail. Refer to Figure 1 The flow schematic diagram of a method for detecting the shape of an object based on multiple cameras shown, the method may include the following steps:

[0020] Step S102, in response to the induction trigger operation of the target material, simultaneously collect images of the target material through multiple cameras with different shooting angles to obtain the original images of the target material collected by each camera.

[0021] Among them, the cameras with different shooting angles are fixedly installed at different target positions corresponding to the target material.

[0022] The target material can be a material placed on a material transmission mechanism (such as a turntable, a conveyor belt). Cameras for collecting material images from different shooting angles are fixedly installed at multiple different positions on the material transmission mechanism. An infrared sensor for sensing the material is also fixedly installed on the material transmission mechanism. Each camera and each infrared sensor are respectively connected to a control unit (such as a PLC, etc.). When the target material is transmitted to the sensing range of the infrared sensor and is sensed by the infrared sensor, after a preset time period (such as a few milliseconds or dozens of milliseconds), multiple cameras with different shooting angles are triggered to simultaneously collect images of the target material, thereby obtaining the original images of the target material collected by each camera.

[0023] Step S104, perform target detection on each original image through a preset target detection model to obtain the detection frames of the target material corresponding to each camera.

[0024] Among them, each detection box has a corresponding first classification confidence level.

[0025] The above-mentioned preset object detection model is a pre-trained object detection model, which can be selected according to actual detection requirements. For example, R-CNN (Region-based Convolutional Neural Networks) series models, YOLO (You Only Look Once) series models, SSD (Single Shot MultiBox Detector) models, etc. are not limited in this regard.

[0026] After obtaining the original images of the target materials collected by each camera, the obtained original images can be input into the preset object detection model for object detection. The object detection results corresponding to each original image are output through the preset object detection model. The object detection result corresponding to each original image includes one or more detection boxes containing the target materials and the classification confidence level of each detection box (i.e., the first classification confidence level, which can be represented by the classification probability belonging to each target).

[0027] Step S106: Perform image segmentation on the region corresponding to the corresponding detection box on each original image to obtain the shape objects of the target material corresponding to each camera.

[0028] After obtaining the detection boxes containing the target materials corresponding to each original image, the duplicate detection boxes can be removed by post-processing the detection boxes corresponding to each original image, and the region within the detection box on each original image can be cropped into a corresponding local image. Then, a traditional edge detection algorithm is used to perform image segmentation on each cropped local image to segment each local image into a shape object region (i.e., the boundary pixel part of the target material) and a non-shape object region (i.e., the part outside the shape object region in the local image), and the pixel part located within the shape object region in all the local images corresponding to each camera is extracted as the shape object of the target material corresponding to that camera, so as to obtain the shape objects of the target material corresponding to each camera.

[0029] Step S108: Compare each shape object with the preset standard shape object respectively to determine the second classification confidence level of each shape object.

[0030] The above-mentioned second classification confidence level characterizes the similarity degree between each shape object and a preset standard shape object. The above-mentioned second classification confidence level can adopt the similarity degree between the corresponding shape object and the preset standard shape object (such as cosine similarity, covariance of pixels, Pearson correlation coefficient of pixels, etc.), distance (such as Euclidean distance, Minkowski distance, Chebyshev distance, Mahalanobis distance, Hamming distance, etc.), etc., which can be selected according to actual needs and is not limited herein.

[0031] Step S110: Determine the shape detection result of the target material based on the first classification confidence level of each detection box and the second classification confidence level of each shape object.

[0032] After obtaining the first classification confidence level of each detection box and the second classification confidence level of each shape object, the first classification confidence level and the second classification confidence level corresponding to the same camera can be screened out according to the camera used to collect the original image, and then the shape detection result of the target material can be analyzed and determined based on the first classification confidence level and the second classification confidence level corresponding to each camera.

[0033] A method for detecting the shape of a material based on multiple cameras provided by an embodiment of the present invention first triggers multiple cameras at different shooting angles to simultaneously collect the original image of the target material through the induction trigger operation of the target material, then performs target detection on each original image through a preset target detection model to obtain a detection box with a first classification confidence level corresponding to the target material for each camera, then performs image segmentation on the area corresponding to the corresponding detection box on each original image to obtain a shape object corresponding to the target material for each camera, and respectively compares each shape object with a preset standard shape object to determine the second classification confidence level of each shape object. Finally, the shape detection result of the target material is determined based on the first classification confidence level of each detection box and the second classification confidence level of each shape object. By adopting the above technology, the target detection can be performed on the material images collected by multiple cameras to obtain the material detection box and its classification confidence level, and the material shape and its classification confidence level can be obtained by using the material detection box through the image segmentation method. Furthermore, the shape detection result of the material can be obtained by combining the classification confidence levels of the material detection box and the material shape respectively, thereby improving the efficiency and accuracy of material classification.

[0034] As a possible implementation manner, the above step S110 (that is, determining the shape detection result of the target material based on the first classification confidence level of each detection box and the second classification confidence level of each shape object) may include:

[0035] Step 1: Determine the third classification confidence level of the target material corresponding to each camera based on the first classification confidence level and the second classification confidence level corresponding to each camera.

[0036] Exemplarily, in the above step 1, for each camera, the first classification confidence and the second classification confidence corresponding to the camera can be added, and the result obtained after the addition is normalized to obtain the third classification confidence corresponding to the camera.

[0037] For example, if the preset object detection model uses yolov5, for the i-th camera, after obtaining the first classification confidence and the second classification confidence corresponding to the camera, the fused classification confidence corresponding to the camera can be calculated according to the following formula: Ron_conf i =(Yolo_conf i +Tra_conf i ), where Ron_conf i is the fused classification confidence corresponding to the i-th camera, Yolo_conf i is the first classification confidence of the detection box corresponding to the i-th camera, and Tra_conf i is the second classification confidence of the shape object corresponding to the i-th camera; then the fused classification confidence corresponding to the camera is normalized, and the result obtained after the normalization is used as the third classification confidence corresponding to the camera.

[0038] Step 2: Based on the shape parameters of each shape object and the third classification confidence corresponding to each camera, determine the shape detection result of the target material.

[0039] Exemplarily, the above step 2 can be performed according to the following operation method:

[0040] Step 21: Based on the shape parameters of each shape object and the third classification confidence corresponding to each camera, determine the first classification weight of the target material corresponding to each camera.

[0041] Step 22: Based on the third classification confidence and the first classification weight corresponding to each camera, determine the overall classification confidence of the target material.

[0042] Step 23: Normalize the overall classification confidence of the target material, and determine whether the result obtained after the normalization meets the preset conditions, and then determine the shape detection result of the target material according to the judgment result.

[0043] As a possible implementation manner, the above shape parameters may include pixel area; based on this, the above step 21 (that is, based on the shape parameters of each shape object and the third classification confidence corresponding to each camera, determine the first classification weight of the target material corresponding to each camera) may include:

[0044] Step A: From the third classification confidence levels corresponding to each camera, count the maximum value as the optimized classification confidence level corresponding to that camera. From the pixel areas of the shape objects corresponding to all cameras, count the average value as the average pixel area of the target material.

[0045] The above average value can be an arithmetic mean, geometric mean, square mean (i.e., root mean square), harmonic mean, weighted mean, etc. Specifically, it can be selected according to actual needs and is not limited in this regard.

[0046] For the calculation of the optimized classification confidence level, for example, if there are two third classification confidence levels corresponding to a certain camera, belonging to the target material and the non-target material (i.e., the environment around the target material) respectively, and the third classification confidence level belonging to the target material is greater than that belonging to the non-target material, then the third classification confidence level belonging to the target material corresponding to this camera can be taken as the optimized classification confidence level corresponding to this camera.

[0047] Step B: Based on the average pixel area, the pixel areas of each shape object, and the optimized classification confidence levels corresponding to each camera, determine the first classification weights corresponding to each camera.

[0048] Exemplarily, in the above Step B, for each camera, based on the average pixel area, the pixel area of the shape object corresponding to this camera, and the optimized classification confidence level corresponding to this camera, the following formula can be used to calculate the first classification weight corresponding to this camera:

[0049] W i =(1 + log(Conf max_i * 10)) + (1 + log(Area i / Area mean ))

[0050] where W i is the first classification weight corresponding to the i-th camera, Conf max_i is the optimized classification confidence level corresponding to the i-th camera, Area i is the pixel area of the shape object corresponding to the i-th camera, and Area mean is the average pixel area.

[0051] As a possible implementation, the above Step 22 (i.e., based on the third classification confidence levels and the first classification weights corresponding to each camera, determine the overall classification confidence level of the target material) may include:

[0052] Step a: Perform truncation processing on the first classification weights corresponding to each camera to obtain the second classification weights corresponding to each camera.

[0053] Continuing from the previous example, after obtaining the first classification weight W corresponding to the i-th camera i , the preset constraint condition W i ∈[0, 3] can be used to further perform a truncation process on W i , that is: if W i is less than 0, then set W i to 0, and if W i is greater than 3, then set W i to 3.

[0054] Step b, for each camera, based on the third classification confidence and the second classification weight corresponding to the camera, calculate the overall classification confidence of the target material using the following formula:

[0055]

[0056] where Conf result is the overall classification confidence, n is the number of cameras, Conf i is the third classification confidence corresponding to the i-th camera, and w i is the second classification weight corresponding to the i-th camera.

[0057] For ease of understanding, the operation method of the above-mentioned multi-camera-based material shape detection method is described exemplarily below with a specific application as an example.

[0058] Refer to Figure 2 as shown. The above-mentioned multi-camera-based material shape detection method can be performed according to the following operation method:

[0059] The overall algorithm flow is as shown in the above figure:

[0060] Step 1, multi-camera photographing.

[0061] The material is placed on a turntable for transportation. There are multiple photographing positions with fixed brackets near the turntable. Multiple cameras are installed on the fixed brackets of the photographing positions one by one at different shooting angles, so that the material can be photographed from multiple angles by multiple cameras; an infrared sensor is also set near the turntable. When the infrared sensor senses the material, it triggers multiple cameras to photograph the material, and each camera obtains a corresponding original material photo.

[0062] Step 2, Yolov5 material detection.

[0063] Perform material detection on each original material photo through the pre-trained Yolov5 algorithm to obtain the bbox (bounding box, also known as the detection box) of each original material photo, and each bbox has a corresponding first classification confidence.

[0064] Step 3, Shape judgment by traditional algorithm.

[0065] Perform NMS (Non-Maximum Suppression) post-processing on the bboxes obtained from Yolov5 material detection, and crop the regions within the remaining bboxes in each original material photo into small-sized images. Perform adaptive threshold segmentation on each small-sized image through a traditional edge algorithm. Then, extract the corresponding material boundary pixels from the segmentation results of each small-sized image as the material shape. After that, calculate the similarity between each material shape and the standard shape as the corresponding second classification confidence.

[0066] Step 4, Calculation of adaptive weights for multiple cameras.

[0067] Based on the first classification confidence and the second classification confidence corresponding to each camera, calculate the fused classification confidence corresponding to each camera according to the following formula: Ron_conf i =(Yolo_conr i +Tra_conf i ); Then, perform normalization processing on the fused classification confidence corresponding to each camera to obtain the material judgment confidence (i.e., the third classification confidence) corresponding to each camera; Calculate the maximum value of the material judgment confidence corresponding to each camera as the corresponding optimized confidence (i.e., the optimized classification confidence); Calculate the pixel area of the material shapes corresponding to all cameras, and calculate the average pixel area of the material shapes corresponding to all cameras; Based on this average pixel area, the pixel area of each material shape, and the optimized confidence corresponding to each camera, calculate the material judgment weight (i.e., the first classification weight) corresponding to each camera according to the following formula: W i =(1 + log(Conf max_i * 10))+(1 + log(Area i / Area mean )); Perform truncation processing on the material judgment weight corresponding to each camera with W i ∈[0, 3] to obtain the truncated weight (i.e., the second classification weight) corresponding to each camera; Based on the material judgment confidence and the truncated weight corresponding to this camera, calculate the weighted confidence of the material (i.e., the overall classification confidence) using the following formula: Then, perform normalization processing on this weighted confidence to obtain the final confidence of the material.

[0068] Step 5, Output the final result.

[0069] Determine whether the final confidence level of the material meets the threshold requirement (i.e., determine whether the final confidence level is greater than the preset threshold). If it meets (i.e., the final confidence level is greater than the preset threshold), output the first result indicating that the shape of the material is qualified as the final result. If it does not meet (i.e., the final confidence level is not greater than the preset threshold), output the second result indicating that the shape of the material is unqualified as the final result.

[0070] Using the above multi-camera-based material shape detection method, compared with the method of manually detecting materials or other methods of directly using vision detection algorithms to detect materials, for materials with irregular shapes and unfixed incoming materials, the weights corresponding to cameras at different shooting angles can be adaptively adjusted, and the shape judgment of the materials is more accurate, especially for materials with complex shapes, the detection effect is better.

[0071] Based on the above multi-camera-based material shape detection method, an embodiment of the present invention further provides a multi-camera-based material shape detection device. Refer to Figure 3 As shown, the device may include the following modules:

[0072] The image acquisition module 302 is configured to, in response to the induction trigger operation of the target material, simultaneously acquire images of the target material through multiple cameras at different shooting angles, and obtain the original images of the target material acquired by each camera; wherein, the cameras at different shooting angles are fixedly installed at different target positions corresponding to the target material.

[0073] The target detection module 304 is configured to perform target detection on each original image through a preset target detection model to obtain the detection frames of the target material corresponding to each camera; wherein, each detection frame has a corresponding first classification confidence level.

[0074] The image segmentation module 306 is configured to perform image segmentation on the region corresponding to the corresponding detection frame on each original image to obtain the shape objects of the target material corresponding to each camera.

[0075] The first determination module 308 is configured to respectively compare each shape object with a preset standard shape object to determine the second classification confidence level of each shape object.

[0076] The second determination module 310 is configured to determine the shape detection result of the target material based on the first classification confidence level of each detection frame and the second classification confidence level of each shape object.

[0077] An object shape detection device based on multiple cameras provided by an embodiment of the present invention can perform object detection on object images collected by multiple cameras to obtain object detection frames and their classification confidence levels, and use the object detection frames to obtain object shapes and their classification confidence levels through image segmentation. Furthermore, by combining the classification confidence levels of the object detection frames and the object shapes respectively, the shape detection result of the object is obtained, thereby improving the efficiency and accuracy of object classification.

[0078] The above-mentioned second determination module 310 can also be used to: based on the first classification confidence level and the second classification confidence level corresponding to each camera, determine the third classification confidence level of the target object corresponding to each camera; based on the shape parameters of each shape object and the third classification confidence level corresponding to each camera, determine the shape detection result of the target object.

[0079] The above-mentioned second determination module 310 can also be used to: for each camera, add the first classification confidence level and the second classification confidence level corresponding to the camera, and perform normalization processing on the obtained result to obtain the third classification confidence level corresponding to the camera.

[0080] The above-mentioned second determination module 310 can also be used to: based on the shape parameters of each shape object and the third classification confidence level corresponding to each camera, determine the first classification weight of the target object corresponding to each camera; based on the third classification confidence level and the first classification weight corresponding to each camera, determine the overall classification confidence level of the target object; perform normalization processing on the overall classification confidence level of the target object, and determine whether the obtained result after normalization processing meets a preset condition, and then determine the shape detection result of the target object according to the determination result.

[0081] The above-mentioned shape parameters can include pixel area; based on this, the above-mentioned second determination module 310 can also be used to: count the maximum value from the third classification confidence levels corresponding to each camera as the optimized classification confidence level corresponding to the camera, and count the average value from the pixel areas of the shape objects corresponding to all cameras as the average pixel area of the target object; based on the average pixel area, the pixel areas of each shape object, and the optimized classification confidence level corresponding to each camera, determine the first classification weight corresponding to each camera.

[0082] The above-mentioned second determination module 310 can also be used to: for each camera, based on the average pixel area, the pixel area of the shape object corresponding to the camera, and the optimized classification confidence level corresponding to the camera, calculate the first classification weight corresponding to the camera using the following formula:

[0083] W i =(1 + log(Conf max_i * 10))+(1 + log(Area i / Areamean ))

[0084] Among them, W i is the first classification weight corresponding to the i-th camera, and Conf max_i is the optimized classification confidence corresponding to the i-th camera, and Area i is the pixel area of the shape object corresponding to the i-th camera, and Area mean is the average pixel area.

[0085] The above-mentioned second determination module 310 can also be used to: perform truncation processing on the first classification weights corresponding to each camera to obtain the second classification weights corresponding to each camera; for each camera, based on the third classification confidence and the second classification weight corresponding to the camera, calculate the overall classification confidence of the target material by using the following formula:

[0086]

[0087] Among them, Conf result is the overall classification confidence, n is the number of cameras, Conf i is the third classification confidence corresponding to the i-th camera, and w i is the second classification weight corresponding to the i-th camera.

[0088] The material shape detection device based on multiple cameras provided by the embodiments of the present invention has the same implementation principle and the same technical effects as those of the foregoing embodiments of the material shape detection method based on multiple cameras. For the sake of brief description, for the parts not mentioned in the device embodiments, reference may be made to the corresponding content in the foregoing method embodiments.

[0089] The embodiments of the present invention also provide an electronic device, as Figure 4 shown, which is a schematic structural diagram of the electronic device. Among them, the electronic device includes a processor 41 and a memory 40. The memory 40 stores computer executable instructions that can be executed by the processor 41, and the processor 41 executes the computer executable instructions to implement the above-mentioned material shape detection method based on multiple cameras.

[0090] In Figure 4 the illustrated embodiment, the electronic device further includes a bus 42 and a communication interface 43. Among them, the processor 41, the communication interface 43, and the memory 40 are connected through the bus 42.

[0091] Among them, the memory 40 may include high-speed random access memory (RAM), and may also include non-volatile memory, such as at least one disk memory. The communication connection between this system network element and at least one other network element is realized through at least one communication interface 43 (which can be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. can be used. The bus 42 can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, an EISA (Extended Industry Standard Architecture) bus, etc. The bus 42 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 only a bidirectional arrow is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0092] The processor 41 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above multi-camera-based material shape detection method can be completed by the integrated logic circuit in the hardware of the processor 41 or instructions in the form of software. The above-mentioned processor 41 can be a general-purpose processor, including a central processing unit (CPU for short), a network processor (NP for short), etc.; it can also be a digital signal processor (DSP for short), an application-specific integrated circuit (ASIC for short), a field-programmable gate array (FPGA for short), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the multi-camera-based material shape detection method disclosed in combination with the embodiments of the present invention can be directly embodied as being executed and completed by a hardware decoding processor, or executed and completed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the art such as random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, registers, etc. This storage medium is located in the memory, and the processor 41 reads the information in the memory and combines its hardware to complete the steps of the multi-camera-based material shape detection method in the foregoing embodiments.

[0093] Unless otherwise specifically stated, the relative steps, numerical expressions, and numerical values of the components and steps set forth in these embodiments do not limit the scope of the present invention.

[0094] If the described functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium executable by a processor. Based on such understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs that can store program codes.

[0095] In the description of the present invention, it should be noted that the orientation or positional relationship indicated by the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. is based on the orientation or positional relationship shown in the drawings. It is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention. In addition, the terms "first", "second", "third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0096] Finally, it should be noted that the above-described embodiments are only specific embodiments of the present invention, used to illustrate the technical solutions of the present invention, and are not intended to limit them. The protection scope of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: any person skilled in the art within the technical scope disclosed by the present invention can still modify the technical solutions described in the foregoing embodiments, or can easily conceive of changes, or perform equivalent replacements for some of the technical features; and these modifications, changes, or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be determined by the protection scope of the claims.

Claims

1. A method for detecting the shape of materials based on multiple cameras, characterized in that, the method includes: In response to the induction trigger operation of the target material, images of the target material are simultaneously collected by multiple cameras with different shooting angles, and the original images of the target material collected by each camera are obtained; wherein, the cameras with different shooting angles are fixedly installed at different target positions corresponding to the target material; The original images are subjected to target detection through a preset target detection model to obtain detection frames of the target material corresponding to each camera; wherein, each detection frame has a corresponding first classification confidence; The area corresponding to the detection frame on each original image is subjected to image segmentation to obtain shape objects of the target material corresponding to each camera; Each shape object is respectively compared with a preset standard shape object to determine the second classification confidence of each shape object; Based on the first classification confidence of each detection frame and the second classification confidence of each shape object, the shape detection result of the target material is determined.

2. The method according to claim 1, characterized in that, determining the shape detection result of the target material based on the first classification confidence of each detection frame and the second classification confidence of each shape object includes: Based on the first classification confidence and the second classification confidence corresponding to each camera, the third classification confidence of the target material corresponding to each camera is determined; Based on the shape parameters of each shape object and the third classification confidence corresponding to each camera, the shape detection result of the target material is determined.

3. The method according to claim 2, characterized in that, determining the third classification confidence of the target material corresponding to each camera based on the first classification confidence and the second classification confidence corresponding to each camera includes: For each camera, the first classification confidence and the second classification confidence corresponding to the camera are added, and the result obtained after addition is normalized to obtain the third classification confidence corresponding to the camera.

4. The method according to claim 2, characterized in that, determining the shape detection result of the target material based on the shape parameters of each shape object and the third classification confidence corresponding to each camera includes: Based on the shape parameters of each shape object and the third classification confidence corresponding to each camera, the first classification weight of the target material corresponding to each camera is determined; Based on the third classification confidence and the first classification weight corresponding to each camera, the overall classification confidence of the target material is determined; The overall classification confidence of the target material is normalized, and it is judged whether the result obtained after normalization meets the preset conditions, and then the shape detection result of the target material is determined according to the judgment result.

5. The method according to claim 4, characterized in that, the shape parameters include pixel area; based on the shape parameters of each shape object and the third classification confidence corresponding to each camera, determining the first classification weight of the target material corresponding to each camera includes: Statistically obtain the maximum value from the third classification confidence levels corresponding to each camera as the optimized classification confidence level corresponding to that camera, and statistically obtain the average value from the pixel areas of the shape objects corresponding to all cameras as the average pixel area of the target material; Based on the average pixel area, the pixel areas of each shape object, and the optimized classification confidence levels corresponding to each camera, determine the first classification weights corresponding to each camera.

6. The method according to claim 5, wherein, Based on the average pixel area, the pixel areas of each shape object, and the optimized classification confidence levels corresponding to each camera, determining the first classification weights corresponding to each camera includes: for each camera, based on the average pixel area, the pixel area of the shape object corresponding to that camera, and the optimized classification confidence level corresponding to that camera, calculate the first classification weight corresponding to that camera using the following formula: W i = (1 + log(Conf max_i * 10)) + (1 + log(Area i / Area mean )) Among them, W i is the first classification weight corresponding to the i-th camera, Conf max_i is the optimized classification confidence corresponding to the i-th camera, Area i is the pixel area of the shape object corresponding to the i-th camera, Area mean is the average pixel area.

7. The method according to claim 6, wherein, Based on the third classification confidence levels and the first classification weights corresponding to each camera, determining the overall classification confidence level of the target material includes: Perform a truncation process on the first classification weights corresponding to each camera to obtain the second classification weights corresponding to each camera; For each camera, based on the third classification confidence level and the second classification weight corresponding to that camera, calculate the overall classification confidence level of the target material using the following formula: Among them, Conf result is the overall classification confidence, n is the number of cameras, and Conf i is the third classification confidence corresponding to the i-th camera, and w i is the second classification weight corresponding to the i-th camera.

8. A material shape detection device based on multiple cameras, wherein, The device includes: An image acquisition module, configured to, in response to an induction trigger operation of a target material, simultaneously acquire images of the target material through multiple cameras with different shooting angles to obtain the original images of the target material corresponding to each camera; wherein, the cameras with different shooting angles are fixedly installed at different target positions corresponding to the target material; A target detection module, configured to perform target detection on each original image through a preset target detection model to obtain the detection frames corresponding to the target material for each camera; wherein, each detection frame has a corresponding first classification confidence level; An image segmentation module, configured to perform image segmentation on the region corresponding to the corresponding detection frame on each original image to obtain the shape objects corresponding to the target material for each camera; A first determination module, configured to respectively compare each shape object with a preset standard shape object to determine the second classification confidence level of each shape object; A second determination module, configured to determine the shape detection result of the target material based on the first classification confidence levels of each detection frame and the second classification confidence levels of each shape object.

9. The device according to claim 8, wherein, The second determination module is further configured to: Based on the first classification confidence level and the second classification confidence level corresponding to each camera, determine the third classification confidence level corresponding to the target material for each camera; Based on the shape parameters of each shape object and the third classification confidence levels corresponding to each camera, determine the shape detection result of the target material.

10. An electronic device, wherein, It includes a processor and a memory, and the memory stores computer-executable instructions that can be executed by the processor. The processor executes the computer-executable instructions to implement the method according to any one of claims 1 to 7.

Images