Multi-camera-based material shape detection method, device and electronic equipment

Abstract

The application provides a multi-camera-based material shape detection method and device and electronic equipment. In response to an induction trigger operation of a target material, multiple cameras with different shooting angles simultaneously perform image acquisition on the target material to obtain original images of the target material collected by each camera. A preset target detection model is used to perform target detection on each original image to obtain a detection frame corresponding to the target material and each camera. An image segmentation is performed on a region on each original image corresponding to the respective detection frame to obtain a shape object corresponding to the target material and each camera. Each shape object is compared with a preset standard shape object to determine a second classification confidence of each shape object. Based on a first classification confidence of each detection frame and a second classification confidence of each shape object, a shape detection result of the target material is determined. The application can improve the efficiency and accuracy of material classification.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to a multi-camera-based material shape detection method, device and electronic equipment. BACKGROUND

[0002] In a traditional material sorting process, foreign matter or poor-quality materials are sorted out by manual or semi-automatic equipment. Due to the limitations of human eye observation, the sorting efficiency and quality cannot be met. Therefore, visual detection technology is introduced to realize the positioning and classification of materials through image recognition, which can improve the sorting efficiency and quality.

[0003] There are mainly two purposes for material classification through visual detection in the prior art: one is to select materials, that is, to quickly identify defective products in the materials and to remove the defective products; the other is to classify and identify a plurality of materials, and then to sort the plurality of materials. When the prior art classifies materials through visual detection, a camera needs to collect material pictures in real time. With the increase of material types and the diversification and irregularization of the shapes, colors and other characteristics of the materials, and the non-fixed material loading position and loading angle on the flow line, the material picture collected by a single camera cannot completely describe the real characteristics of the material, resulting in low accuracy of material classification determination. The material pictures collected by multiple cameras from different angles are very different in material characteristics, which makes it difficult to ensure the efficiency and accuracy of material classification. SUMMARY

[0004] In view of this, the purpose of the present application is to provide a multi-camera-based material shape detection method, device and electronic equipment to improve the efficiency and accuracy of material classification, thereby alleviating the above-mentioned problems in the related art.

[0005] In a first aspect, an embodiment of the present application provides a multi-camera-based material shape detection method, which comprises: in response to an induction trigger operation of a target material, simultaneously collecting images of the target material through a plurality of cameras with different shooting angles to obtain original images of the target material collected by each camera; wherein the cameras with different shooting angles are fixedly installed at different target positions corresponding to the target material; performing target detection on each original image through a preset target detection model to obtain a detection frame corresponding to the target material for each camera; wherein each detection frame has a corresponding first classification confidence; performing image segmentation on a region corresponding to the corresponding detection frame on each original image to obtain a shape object corresponding to the target material for each camera; comparing each shape object with a preset standard shape object to determine a second classification confidence of each shape object; and determining a shape detection result of the target material based on the first classification confidence of each detection frame and the second classification confidence of each shape object.

[0006] In a second aspect, the embodiments of the present application also provide a multi-camera-based material shape detection device, which comprises: an image acquisition module, configured to acquire images of a target material through a plurality of cameras with different shooting angles simultaneously in response to an inductive trigger operation of the target material, to obtain original images of the target material acquired by each camera; wherein the cameras with different shooting angles are fixedly installed at different target positions corresponding to the target material; a target detection module, configured to detect targets in each original image through a preset target detection model, to obtain detection boxes of the target material corresponding to each camera; wherein each detection box has a corresponding first classification confidence; an image segmentation module, configured to segment an area on each original image corresponding to the corresponding detection box, to obtain shape objects of the target material corresponding to each camera; a first determination module, configured to compare each shape object with a preset standard shape object respectively to determine a second classification confidence of each shape object; and a second determination module, configured to determine a shape detection result of the target material based on the first classification confidence of each detection box and the second classification confidence of each shape object.

[0007] In a third aspect, the embodiments of the present application also provide an electronic device, which comprises a processor and a memory, the memory stores computer executable instructions capable of being executed by the processor, and the processor executes the computer executable instructions to implement the multi-camera-based material shape detection method of the first aspect.

[0008] The multi-camera-based material shape detection method, device and electronic device provided by the embodiments of the present application first trigger a plurality of cameras with different shooting angles to acquire original images of a target material simultaneously through an inductive trigger operation of the target material, then detect targets in each original image through a preset target detection model to obtain detection boxes of the target material corresponding to each camera with a first classification confidence, then segment an area on each original image corresponding to the corresponding detection box to obtain shape objects of the target material corresponding to each camera, compare each shape object with a preset standard shape object respectively to determine a second classification confidence of each shape object, and finally determine a shape detection result of the target material based on the first classification confidence of each detection box and the second classification confidence of each shape object. By using the above technology, the material images acquired by the plurality of cameras are used for target detection to obtain material detection boxes and their classification confidences, the material detection boxes are used for image segmentation to obtain material shapes and their classification confidences, and then the shape detection result of the material is obtained by combining the classification confidences of the material detection boxes and the material shapes, thereby improving the efficiency and accuracy of material classification.

[0009] Other features and advantages of the present application will be set forth in the descriptions that follow, and in part will be apparent from the description, or can be learned by practice of the application. The purposes and other advantages of the present application will be realized and attained by the structures particularly pointed out in the description, claims and drawings.

[0010] In order to make the above objectives, features and advantages of the present application more apparent, the following will describe a preferred embodiment in detail, and the accompanying drawings will be referred to, as follows. BRIEF DESCRIPTION OF DRAWINGS

[0011] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the following will briefly introduce the drawings needed to be used in the specific embodiments or the prior art description. Obviously, the drawings described below are some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0012] Figure 1 A flowchart of a material shape detection method based on multiple cameras in an embodiment of the present application;

[0013] Figure 2 A flowchart of a material shape detection method based on multiple cameras in an embodiment of the present application;

[0014] Figure 3 A structural diagram of a material shape detection device based on multiple cameras in an embodiment of the present application;

[0015] Figure 4 A structural diagram of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0016] In order to make the objectives, technical solutions and advantages of the embodiments of the present application more apparent, the technical solutions of the present application will be described clearly and completely in combination with embodiments. Obviously, the described embodiments are some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the present application.

[0017] Currently, existing technologies for material classification using visual inspection primarily serve two purposes: firstly, to select materials, i.e., to quickly identify and remove defective products; and secondly, to classify and identify multiple materials before sorting them. Existing technologies for material classification using visual inspection require cameras to capture material images in real time. However, with the increasing variety and irregularities in material types, shapes, colors, and other characteristics, and the variable loading positions and angles of materials on the production line, images captured by a single camera cannot fully describe the true characteristics of the material, resulting in low accuracy in material classification. Furthermore, images captured by multiple cameras from different angles, due to significant differences in material characteristics, make it difficult to guarantee both efficiency and accuracy in material classification.

[0018] Based on this, the present invention provides a material shape detection method, device and electronic device based on multiple cameras, which can improve the efficiency and accuracy of material classification, thereby alleviating the above-mentioned problems existing in related technologies.

[0019] To facilitate understanding of this embodiment, a multi-camera-based material shape detection method disclosed in this invention will first be described in detail. (See [link to relevant documentation]). Figure 1 The diagram shows a flowchart of a material shape detection method based on multiple cameras. The method may include the following steps:

[0020] In step S102, in response to the sensing trigger operation of the target material, multiple cameras with different shooting angles simultaneously acquire images of the target material to obtain the original images of the target material acquired by each camera.

[0021] In this setup, cameras with different shooting angles are fixedly installed at different target positions corresponding to the target material.

[0022] The target material can be placed on a material conveying mechanism (such as a turntable or conveyor belt). Multiple cameras are fixedly installed at different positions on the material conveying mechanism to capture images of the material from different shooting angles. Infrared sensors for sensing the material are also fixedly installed on the material conveying mechanism. Each camera and each infrared sensor is connected to a control unit (such as a PLC). When the target material is transmitted into the sensing range of the infrared sensor and is sensed by the infrared sensor, the control unit triggers multiple cameras at different shooting angles to simultaneously capture images of the target material after a preset time (such as a few milliseconds or tens of milliseconds), thereby obtaining the original images of the target material captured by each camera.

[0023] Step S104: Target detection is performed on each original image using a preset target detection model to obtain the detection boxes corresponding to the target materials for each camera.

[0024] Each of the detection boxes has a corresponding first classification confidence.

[0025] The preset target detection model is a pre-trained target detection model, which can be selected according to actual detection requirements, such as R-CNN (Region-based Convolutional Neural Networks) series model, YOLO (You Only Look Once) series model, SSD (Single Shot MultiBox Detector) model, etc.

[0026] After obtaining the original images of the target materials collected by each camera, the obtained original images are input into the preset target detection model for target detection, and the preset target detection model outputs the target detection results corresponding to each original image. Each target detection result corresponding to each original image includes one or more detection boxes containing target materials and the classification confidence (i.e. the first classification confidence, which can be represented by the classification probability belonging to each target) of each detection box.

[0027] In step S106, the region corresponding to each detection box on each original image is segmented to obtain the shape object of the target material corresponding to each camera.

[0028] After obtaining the detection boxes containing target materials corresponding to each original image, the detection boxes corresponding to each original image are processed to remove repeated detection boxes, and the region within the detection box on each original image is cropped into a corresponding local image. Then, a traditional edge detection algorithm is used to segment each local image into a shape object region (i.e. the boundary pixel part of the target material) and a non-shape object region (i.e. the part outside the shape object region in the local image), and the pixel part within the shape object region in all local images corresponding to each camera is extracted as the shape object of the target material corresponding to the camera, thereby obtaining the shape object of the target material corresponding to each camera.

[0029] In step S108, each shape object is compared with a preset standard shape object to determine the second classification confidence of each shape object.

[0030] The second classification confidence degree represents a similarity degree between each shape object and a preset standard shape object. The second classification confidence degree can adopt a similarity (such as a cosine similarity, a pixel covariance, a pixel Pearson correlation coefficient, etc.), a distance (such as a Euclidean distance, a Mahalanobis distance, a Chebyshev distance, a Mahalanobis distance, a Hamming distance, etc.), etc. between the corresponding shape object and the preset standard shape object. The second classification confidence degree can be selected according to actual needs, and is not limited.

[0031] In step S110, the shape detection result of the target material is determined based on the first classification confidence degrees of the detection frames and the second classification confidence degrees of the shape objects.

[0032] After obtaining the first classification confidence degrees of the detection frames and the second classification confidence degrees of the shape objects, the first classification confidence degrees and the second classification confidence degrees corresponding to the same camera can be screened out according to the camera used to collect the original images, and then the shape detection result of the target material is determined based on the first classification confidence degrees and the second classification confidence degrees corresponding to each camera.

[0033] The shape detection method based on multiple cameras provided by the embodiment of the present application first triggers multiple cameras with different shooting angles to simultaneously collect original images of the target material through an inductive trigger operation of the target material, then performs target detection on each original image through a preset target detection model to obtain a detection frame corresponding to each camera and having a first classification confidence degree, then performs image segmentation on a region corresponding to the detection frame on each original image to obtain a shape object corresponding to each camera, and then compares each shape object with a preset standard shape object to determine a second classification confidence degree of each shape object. Finally, the shape detection result of the target material is determined based on the first classification confidence degrees of the detection frames and the second classification confidence degrees of the shape objects. By using the above technology, the target detection can be performed on the material images collected by multiple cameras to obtain a material detection frame and a classification confidence degree thereof, the material shape and a classification confidence degree thereof can be obtained through image segmentation by using the material detection frame, and then the shape detection result of the material is obtained by combining the classification confidence degrees of the material detection frame and the material shape, thereby improving the efficiency and accuracy of material classification.

[0034] As a possible implementation, the step S110 (i.e., determining the shape detection result of the target material based on the first classification confidence degrees of the detection frames and the second classification confidence degrees of the shape objects) can include:

[0035] In step 1, a third classification confidence degree corresponding to each camera of the target material is determined based on the first classification confidence degree and the second classification confidence degree corresponding to each camera.

[0036] Exemplarily, in step 1, for each camera, the first classification confidence and the second classification confidence corresponding to the camera can be added, and the result after addition is normalized to obtain the third classification confidence corresponding to the camera.

[0037] For example, the preset target detection model adopts yolov5, and for the i th camera, after obtaining the first classification confidence and the second classification confidence corresponding to the camera, the fusion classification confidence corresponding to the camera can be calculated according to the following formula: Ron_conf i = (Yolo_conf i + Tra_conf i ), wherein Ron_conf i is the fusion classification confidence corresponding to the i th camera, Yolo_conf i is the first classification confidence of the detection frame corresponding to the i th camera, and Tra_conf i is the second classification confidence of the shape object corresponding to the i th camera; and the fusion classification confidence corresponding to the camera is normalized, and the result after normalization is taken as the third classification confidence corresponding to the camera.

[0038] In step 2, the shape detection result of the target material is determined based on the shape parameters of each shape object and the third classification confidence corresponding to each camera.

[0039] Exemplarily, step 2 can be performed according to the following operation mode:

[0040] In step 21, the first classification weight of the target material corresponding to each camera is determined based on the shape parameters of each shape object and the third classification confidence corresponding to each camera.

[0041] In step 22, the overall classification confidence of the target material is determined based on the third classification confidence and the first classification weight corresponding to each camera.

[0042] In step 23, the overall classification confidence of the target material is normalized, and it is judged whether the result after normalization meets a preset condition, and then the shape detection result of the target material is determined according to the judgment result.

[0043] As a possible implementation, the shape parameters can include a pixel area, and based on this, step 21 (i.e., determining the first classification weight of the target material corresponding to each camera based on the shape parameters of each shape object and the third classification confidence corresponding to each camera) can include:

[0044] Step A, the maximum value of the third classification confidence corresponding to each camera is taken as the optimization classification confidence corresponding to the camera, and the average value of the pixel area of the shape object corresponding to all cameras is taken as the average pixel area of the target material.

[0045] The average value can be an arithmetic mean, a geometric mean, a quadratic mean (i.e. root mean square), a harmonic mean, a weighted mean, etc. The specific selection is not limited.

[0046] For the calculation of the optimization classification confidence, for example, the third classification confidence corresponding to a camera has two, which belong to the target material and the non-target material (i.e. the environment around the target material) respectively, and the third classification confidence corresponding to the target material is greater than the third classification confidence corresponding to the non-target material, so the third classification confidence corresponding to the target material corresponding to the camera can be taken as the optimization classification confidence corresponding to the camera.

[0047] Step B, based on the average pixel area, the pixel area of each shape object and the optimization classification confidence corresponding to each camera, the first classification weight corresponding to each camera is determined.

[0048] Exemplarily, in the above step B, for each camera, the first classification weight corresponding to the camera can be calculated based on the average pixel area, the pixel area of the shape object corresponding to the camera and the optimization classification confidence corresponding to the camera, using the following formula:

[0049] W i i = (1+log(Conf max_i i *10))+(1+log(Area i i / Area mean ))

[0050] Wherein, W i is the first classification weight corresponding to the i-th camera, Conf max_i is the optimization classification confidence corresponding to the i-th camera, Area i is the pixel area of the shape object corresponding to the i-th camera, and Area mean is the average pixel area.

[0051] As a possible implementation, the above step 22 (i.e. determining the overall classification confidence of the target material based on the third classification confidence and the first classification weight corresponding to each camera) can include:

[0052] Step a, the first classification weight corresponding to each camera is truncated to obtain the second classification weight corresponding to each camera.

[0053] After the foregoing example, the first classification weight W i is obtained, and the first classification weight W i is obtained, and the first classification weight W i is truncated, that is, if W i is less than 0, W i is set to 0, and if W i is greater than 3, W i is set to 3.

[0054] Step b, for each camera, based on the third classification confidence and the second classification weight corresponding to the camera, the overall classification confidence of the target material is calculated using the following formula:

[0055]

[0056] wherein Conf result is the overall classification confidence, n is the number of cameras, Conf i is the third classification confidence corresponding to the i-th camera, and w i is the second classification weight corresponding to the i-th camera.

[0057] For ease of understanding, the operation mode of the above-mentioned material shape detection method based on multiple cameras is described as follows by taking a specific application as an example.

[0058] Referring to Figure 2 , the above-mentioned material shape detection method based on multiple cameras can be operated in the following mode:

[0059] The overall algorithm flow is shown in the above figure:

[0060] Step 1, multiple cameras take pictures.

[0061] The material is placed on the turntable for conveying, multiple photographing positions with fixed supports are arranged near the turntable, multiple cameras are one-to-one installed on the fixed supports of the photographing positions at different shooting angles, so that the multiple cameras can take pictures of the material from multiple angles; an infrared sensor is also arranged near the turntable, and the infrared sensor triggers the multiple cameras to take pictures of the material when the infrared sensor senses the material, and each camera corresponds to a corresponding raw material photo obtained by taking pictures.

[0062] Step 2, Yolov5 material detection.

[0063] Each raw material photo is subjected to material detection by a pre-trained Yolov5 algorithm to obtain the bbox (bounding box, i.e., detection frame) of each raw material photo, and each bbox has a corresponding first classification confidence.

[0064] Step 3, shape judgment by traditional algorithm.

[0065] After NMS (non-maximum suppression) post-processing of the bbox obtained by Yolov5 material detection, the area in each original material photo located within the bbox retained is cropped into a small size image. Each small size image is subjected to adaptive threshold segmentation by a traditional edge algorithm. Then, the corresponding material boundary pixel part is extracted from the segmentation result of each small size image as the material shape. Then, the similarity between each material shape and the standard shape is calculated as the corresponding second classification confidence.

[0066] Step 4, adaptive weight calculation of multi-camera.

[0067] Based on the first classification confidence and the second classification confidence corresponding to each camera, the fusion classification confidence corresponding to each camera is calculated according to the following formula: Ron_conf i = (Yolo_conr i + Tra_conf i ); the fusion classification confidence corresponding to each camera is normalized to obtain the material judgment confidence (i.e. third classification confidence) corresponding to each camera; the maximum value of the material judgment confidence corresponding to each camera is calculated as the corresponding optimization confidence (i.e. optimization classification confidence); the pixel area of the material shape corresponding to all cameras is calculated, and the average pixel area of the material shape corresponding to all cameras is calculated; based on the average pixel area, the pixel area of each material shape, and the optimization confidence corresponding to each camera, the material judgment weight (i.e. first classification weight) corresponding to each camera is calculated according to the following formula: W i = (1 + log (Conf max_i * 10)) + (1 + log (Area i / Area mean )); the material judgment weight corresponding to each camera is truncated to W i ∈ [0, 3] to obtain the truncated weight (i.e. second classification weight) corresponding to each camera; based on the material judgment confidence and the truncated weight corresponding to the camera, the weighted confidence (i.e. overall classification confidence) of the material is calculated using the following formula: The weighted confidence is normalized to obtain the final confidence of the material.

[0068] Step 5, output final result.

[0069] determining whether the final confidence degree of the material meets a threshold requirement (i.e., whether the final confidence degree is greater than a preset threshold), and outputting a first result representing that the shape of the material is qualified as a final result if the final confidence degree meets the threshold requirement (i.e., whether the final confidence degree is greater than the preset threshold), or outputting a second result representing that the shape of the material is unqualified as the final result if the final confidence degree does not meet the threshold requirement (i.e., whether the final confidence degree is greater than the preset threshold).

[0070] Compared with the manual detection of the material or other direct visual detection algorithm for the material, the above-mentioned multi-camera-based material shape detection method can adaptively adjust the weight corresponding to the camera with different shooting angles, and is more accurate in judging the shape of the material, especially for the material with a complex shape.

[0071] Based on the above-mentioned multi-camera-based material shape detection method, the embodiment of the present application further provides a multi-camera-based material shape detection device, as shown in Figure 3 The device can include the following modules:

[0072] The image acquisition module 302 is configured to acquire images of the target material through a plurality of cameras with different shooting angles simultaneously in response to a sensing trigger operation of the target material, to obtain original images of the target material acquired by each camera; wherein the cameras with different shooting angles are fixedly installed at different target positions corresponding to the target material.

[0073] The target detection module 304 is configured to detect targets in each original image through a preset target detection model to obtain a detection frame corresponding to each camera for the target material; wherein each detection frame has a corresponding first classification confidence degree.

[0074] The image segmentation module 306 is configured to segment the region corresponding to the corresponding detection frame on each original image to obtain a shape object corresponding to each camera for the target material.

[0075] The first determination module 308 is configured to compare each shape object with a preset standard shape object to determine a second classification confidence degree of each shape object.

[0076] The second determination module 310 is configured to determine a shape detection result of the target material based on the first classification confidence degree of each detection frame and the second classification confidence degree of each shape object.

[0077] The embodiment of the present application provides a multi-camera-based material shape detection device, which can utilize material images collected by multiple cameras to perform target detection to obtain a material detection frame and a classification confidence thereof, utilize the material detection frame to obtain a material shape and a classification confidence thereof through image segmentation, and combine the classification confidence of the material detection frame and the classification confidence of the material shape to obtain a shape detection result of the material, so that the efficiency and accuracy of material classification are improved.

[0078] The second determination module 310 can also be configured to: determine, based on the first classification confidence and the second classification confidence corresponding to each camera, a third classification confidence of the target material corresponding to each camera; and determine, based on the shape parameters of the shape objects and the third classification confidence corresponding to each camera, a shape detection result of the target material.

[0079] The second determination module 310 can also be configured to: for each camera, add the first classification confidence and the second classification confidence corresponding to the camera, and perform normalization processing on the result obtained after the addition to obtain the third classification confidence corresponding to the camera.

[0080] The second determination module 310 can also be configured to: determine, based on the shape parameters of the shape objects and the third classification confidence corresponding to each camera, a first classification weight of the target material corresponding to each camera; determine, based on the third classification confidence and the first classification weight corresponding to each camera, an overall classification confidence of the target material; perform normalization processing on the overall classification confidence of the target material, and determine whether the result obtained after the normalization processing meets a preset condition, and then determine the shape detection result of the target material according to the determination result.

[0081] The shape parameters can include a pixel area, and based on this, the second determination module 310 can also be configured to: from the third classification confidence corresponding to each camera, count a maximum value as an optimized classification confidence corresponding to the camera, and from the pixel areas of the shape objects corresponding to all cameras, count an average value as an average pixel area of the target material; and based on the average pixel area, the pixel areas of the shape objects and the optimized classification confidence corresponding to each camera, determine the first classification weight corresponding to each camera.

[0082] The second determination module 310 can also be configured to: for each camera, based on the average pixel area, the pixel area of the shape object corresponding to the camera and the optimized classification confidence corresponding to the camera, calculate the first classification weight corresponding to the camera by using the following formula:

[0083] W i = (1 + log (Conf max_i * 10)) + (1 + log (Area i / Areamean ))

[0084] wherein, W i is the first classification weight corresponding to the i-th camera, Conf max_i is the optimized classification confidence corresponding to the i-th camera, Area i is the pixel area of the shape object corresponding to the i-th camera, Area mean is the average pixel area.

[0085] The second determination module 310 can also be configured to: perform truncation processing on the first classification weights corresponding to the cameras to obtain second classification weights corresponding to the cameras; and for each camera, calculate an overall classification confidence of the target material based on the third classification confidence and the second classification weight corresponding to the camera, using the following formula:

[0086]

[0087] wherein, Conf result is the overall classification confidence, n is the number of cameras, Conf i is the third classification confidence corresponding to the i-th camera, w i is the second classification weight corresponding to the i-th camera.

[0088] The multi-camera-based material shape detection device provided in the embodiments of the present application has the same implementation principle and technical effects as the multi-camera-based material shape detection method described above. For brevity, the part of the device embodiments not mentioned in the description can be referred to the corresponding content in the method embodiments described above.

[0089] The embodiments of the present application also provide an electronic device, as shown in FIG. 13, which is a structural schematic diagram of the electronic device. The electronic device includes a processor 41 and a memory 40. The memory 40 stores computer executable instructions that can be executed by the processor 41. The processor 41 executes the computer executable instructions to implement the multi-camera-based material shape detection method described above. Figure 4 In the embodiment shown in FIG. 13, the electronic device further includes a bus 42 and a communication interface 43. The processor 41, the communication interface 43 and the memory 40 are connected through the bus 42.

[0090] Figure 4

[0091] ​​The memory 40 can include a high-speed random access memory (RAM) and can also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 43 (which can be wired or wireless), and the Internet, a wide area network, a local area network, a metropolitan area network, etc. can be used. The bus 42 can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, an EISA (Extended Industry Standard Architecture) bus, etc. The bus 42 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one bidirectional arrow is used to represent the system network element and at least one other network element, but it does not mean that there is only one bus or one type of bus.

[0092] The processor 41 can be an integrated circuit chip with signal processing capability. In the implementation process, the steps of the above-described multi-camera-based material shape detection method can be completed by the integrated logic circuit of hardware in the processor 41 or the instructions in the form of software. The processor 41 described above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the multi-camera-based material shape detection method disclosed in combination with the embodiments of the present application can be directly embodied as hardware decoding processor execution completion, or executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium in the art. The storage medium is located in the memory, and the processor 41 reads the information in the memory, and combines the hardware to complete the steps of the above-described multi-camera-based material shape detection method.

[0093] Unless specifically stated otherwise, the relative steps, numerical expressions, and numerical values set forth in the examples herein are not intended to limit the scope of the application.

[0094] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a nonvolatile computer readable storage medium executable by a processor. Based on this understanding, the technical solutions of the present application or the part of the prior art that essentially contributes to the prior art or the part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0095] In the description of the present application, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application. In addition, the terms "first", "second", "third" are only for the purpose of description, and cannot be understood as indicating or implying relative importance.

[0096] Finally, it should be noted that: the above-described embodiments are only specific embodiments of the present application, which are used to illustrate the technical solutions of the present application, and are not limited thereto, and the protection scope of the present application is not limited thereto. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can make modifications or easily think of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed by the present application, or make equivalent replacements to some of the technical features; and these modifications, changes or replacements do not cause the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A material shape detection method based on multiple cameras, characterized in that, The method includes: In response to the sensing trigger operation of the target material, multiple cameras with different shooting angles simultaneously capture images of the target material to obtain the original image of the target material captured by each camera; wherein, the cameras with different shooting angles are fixedly installed at different target positions corresponding to the target material. The original images are subjected to target detection by a preset target detection model to obtain the detection boxes corresponding to the target materials of each camera; each detection box has its own first classification confidence score. Image segmentation is performed on the region corresponding to the corresponding detection box in each original image to obtain the shape object of the target material corresponding to each camera; Each shape object is compared with a preset standard shape object to determine the second category confidence level of each shape object; Based on the first category confidence of each detection box and the second category confidence of each shape object, the shape detection result of the target material is determined; Based on the first category confidence of each detection frame and the second category confidence of each shape object, the shape detection result of the target material is determined, including: based on the first category confidence and the second category confidence corresponding to each camera, the third category confidence of the target material corresponding to each camera is determined; based on the shape parameters of each shape object and the third category confidence corresponding to each camera, the shape detection result of the target material is determined.

2. The method according to claim 1, characterized in that, Based on the first and second category confidence scores corresponding to each camera, the third category confidence score for the target material corresponding to each camera is determined, including: For each camera, the first category confidence score and the second category confidence score corresponding to that camera are added together, and the result of the addition is normalized to obtain the third category confidence score corresponding to that camera.

3. The method according to claim 1, characterized in that, Based on the shape parameters of each object and the third-class confidence score of each camera, the shape detection results of the target material are determined, including: Based on the shape parameters of each object and the third-class confidence level of each camera, the first-class weight of the target material corresponding to each camera is determined. Based on the third category confidence score and the first category weight corresponding to each camera, the overall classification confidence score of the target material is determined; The overall classification confidence of the target material is normalized, and it is determined whether the result obtained after normalization meets the preset conditions. Then, the shape detection result of the target material is determined based on the judgment result.

4. The method according to claim 3, characterized in that, The shape parameters include pixel area; based on the shape parameters of each shape object and the third-class confidence level corresponding to each camera, the first-class weight of the target material corresponding to each camera is determined, including: The maximum value is calculated from the third category confidence scores corresponding to each camera as the optimized category confidence score corresponding to that camera, and the average value is calculated from the pixel areas of the shape objects corresponding to all cameras as the average pixel area of ​​the target material. Based on the average pixel area, the pixel area of ​​each shape object, and the optimized classification confidence of each camera, the first classification weight corresponding to each camera is determined.

5. The method according to claim 4, characterized in that, Based on the average pixel area, the pixel area of ​​each shape object, and the optimized classification confidence score corresponding to each camera, the first classification weight corresponding to each camera is determined, including: for each camera, based on the average pixel area, the pixel area of ​​the shape object corresponding to the camera, and the optimized classification confidence score corresponding to the camera, the first classification weight corresponding to the camera is calculated using the following formula: in, For the first The first category weight corresponding to each camera For the first The optimized classification confidence score for each camera. For the first The pixel area of ​​the shape object corresponding to each camera. This represents the average pixel area.

6. The method according to claim 5, characterized in that, Based on the third-class confidence score and the first-class weight corresponding to each camera, the overall classification confidence score of the target material is determined, including: The first classification weights corresponding to each camera are truncated to obtain the second classification weights corresponding to each camera. For each camera, based on the third-class confidence score and the second-class weight corresponding to that camera, the overall classification confidence score of the target material is calculated using the following formula: in, For the overall classification confidence level, For the number of cameras, For the first The third category confidence score corresponding to each camera For the first The second classification weight corresponding to each camera.

7. A material shape detection device based on multiple cameras, characterized in that, The device includes: The image acquisition module is used to respond to the sensing trigger operation of the target material by simultaneously acquiring images of the target material through multiple cameras with different shooting angles, and obtaining the original image of the target material acquired by each camera; wherein, the cameras with different shooting angles are fixedly installed at different target positions corresponding to the target material. The target detection module is used to perform target detection on each original image using a preset target detection model, and obtain the detection boxes corresponding to the target materials for each camera; wherein, each detection box has its own corresponding first classification confidence score; The image segmentation module is used to segment the region corresponding to the corresponding detection box on each original image to obtain the shape object of the target material corresponding to each camera; The first determination module is used to compare each shape object with a preset standard shape object to determine the second classification confidence level of each shape object; The second determination module is used to determine the shape detection result of the target material based on the first classification confidence of each detection box and the second classification confidence of each shape object; The second determining module is further configured to: determine the third category confidence level of the target material corresponding to each camera based on the first category confidence level and the second category confidence level corresponding to each camera; and determine the shape detection result of the target material based on the shape parameters of each shape object and the third category confidence level corresponding to each camera.

8. An electronic device, characterized in that, The method includes a processor and a memory, the memory storing computer-executable instructions that can be executed by the processor, the processor executing the computer-executable instructions to implement the method of any one of claims 1 to 6.

Images