A method for realizing wide-range article identification by using multiple cameras

Abstract

The application discloses a kind of methods for realizing large-scale article identification using multiple cameras, it is related to the image recognition technical field of artificial intelligence, the present application includes the following steps: S1 is set on the identified area frame body, the lower end of the frame body is equipped with connecting assembly, connecting assembly lower end is equipped with a plurality of acquisition camera S2: the acquisition camera is first parameterized calibration, then color calibration is carried out, S3: each pair of adjacent acquisition camera is binocular correction;S4: each acquisition camera image is identified;S5: for each acquisition camera identification result, filter out the identification result of low threshold S6: for each pair of adjacent camera overlap area, result fitting is carried out, S7: for the article identified, judge whether there is stacking completion article identification.The present application is a kind of methods for realizing large-scale article identification using multiple cameras, high recognition accuracy, and can effectively identify some obscured and stacked articles.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image recognition technology of artificial intelligence, and particularly relates to a method for realizing wide-range article recognition by using multiple cameras. BACKGROUND

[0002] With the continuous development of science and technology, robots have been applied in many fields, such as the medical field, the industrial field, the educational field, etc. In particular, in the industrial field, industrial robots grasp industrial parts and then complete processes such as workpiece processing and part assembly on the assembly line. In the robot workflow, the maturity of the robot grasping technology is crucial. Most of the autonomous grasping of robots is controlled based on vision technology, but in a complex environment, the effect of recognition and positioning is greatly reduced, thereby reducing the efficiency of grasping. Therefore, providing accurate coordinates of target articles is a problem that needs to be solved urgently at present, and therefore a wide-range article position recognition method needs to be used.

[0003] In the prior art, computer image recognition technology, in particular artificial intelligence technology based on deep learning neural networks, is used to recognize objects within the imaging range of an ordinary RGB camera, and has been widely applied in many industries. However, there are still three major problems to be solved for many commercial scenarios. In the prior art, one technical route is to use a wide-angle (fisheye) camera to see as much of the required recognition range as possible. However, this preventive approach has the following problems: poor scalability, if the required recognition range exceeds the imaging range of the wide-angle camera, this technical route still does not meet the requirements; the wide-angle camera imaging object is a distorted graph, especially when it is at the edge of the image, the distortion is very obvious, which greatly affects the accuracy of recognition. The common means to improve recognition accuracy is to label more distorted graphs and retrain the neural network, which is labor-intensive; another technical route is to use multiple cameras to work together to ensure that all recognition areas can be seen by at least one camera, and the computer software will unify the recognition results of all cameras to form the final output. The difficulty of this approach is how to accurately fit the recognition results of all cameras. For example, as shown in Figure 7 , if the repeated results are removed, Figure 7 , since the low objects and high objects are recognized by two cameras, the repeated recognition results need to be removed during fitting. Due to the angle of the camera, some (low) objects may be blocked by other (high) objects in a certain direction. For the problem of how to recognize such blocked objects, the technical solution based on a single camera does not have an effective method, but by using multiple cameras installed at different angles, the blocked objects can be seen and thus recognized, for example Figure 8As shown, the low object is blocked by the high object in the camera N imaging, if only a single camera N is used for identification, the low object cannot be identified. But if camera M and camera N are used for identification at the same time, since camera M can see the low object completely, the identification results of the two cameras are correctly fitted, and finally the low object and the high object can be accurately identified at the same time. If some objects are completely blocked (stacked) by other objects directly above, how to identify the blocked objects or warn that the objects are blocked, for such problems, neither the single camera based identification technology nor the multi-camera based identification technology has a particularly effective identification method, so it is necessary to propose a new solution.

[0004] The existing multi-camera implementation large range object identification method has the problems of poor identification effect, unable to accurately identify the blocked objects, and unable to accurately identify the stacked objects. Therefore, a method for realizing large range object identification by using multi-camera is proposed. SUMMARY

[0005] The main purpose of the present application is to provide a method for realizing large range object identification by using multi-camera, which can effectively solve the problems in the background art.

[0006] To achieve the above purpose, the technical scheme adopted by the present application is as follows: a method for realizing large range object identification by using multi-camera, comprising the following steps:

[0007] S1: setting a frame on the identified area, the lower end of the frame is provided with a connecting component, a plurality of connecting components are distributed at equal intervals in an array, and the lower end of the connecting component is provided with a plurality of collection cameras, this step ensures that each identification area is photographed by at least two adjacent cameras;

[0008] S2: first performing parameter calibration on the collection camera, and then performing color calibration,

[0009] S3: performing binocular correction on each pair of adjacent collection cameras;

[0010] S4: performing object identification on each collection camera image; inputting the image of each collection camera into an artificial intelligence image recognition network on a computer to obtain all the objects identified in the image, for the identified object i, it is expressed as Oi =(ci,pi,xi,yi,wi,hi), wherein: ci represents the object category, pi represents the identification confidence, xi and yi are the coordinates of the upper left corner of the object rectangular frame, and wi and hi are the width and height of the object rectangular frame;

[0011] S5: For each of the identification results of the collection camera, filter out the low-confidence items, if pi < confidence threshold, delete the item from the identification result, and thus obtain a set of identification results for each of the collection cameras C ;

[0012] S6: For each pair of adjacent cameras, perform result fitting, and the specific steps of fitting are as follows:

[0013] S601: For any object Oi recognized by the collection camera C, calculate the region A where Oi is located, and the corresponding adjacent collection camera C imaging region is divided into 4 parts: upper left A0, lower left A1, upper right A2 and lower right A3, each region and the front and rear adjacent collection camera overlap, calculate the overlap coefficient γ of Oi and region Ak (k = 0, 1, 2, 3) The formula is: , and the formula for calculating the region where Oi is located is, From the above formula, if the overlap coefficient of Oi and Ak is the largest, it is considered that Oi belongs to this region Ak;

[0014] S602: When Oi belongs to region A, for each camera C' adjacent to A, obtain all the items of class ci recognized in region A, denoted as ', the discrimination of the identification result of C' whether belongs to A is calculated by the formula in step S601, and then find the items of class ci also recognized in region A by the adjacent collection camera, this step is mainly used to confirm whether it is repeated identification in the next step;

[0015] S603: Calculate the best match of Oi and ' all recognized items, for the identification result Oj in ', the matching calculation of Oi and Oj is: , the best matching item of Oi and ' is: , if the best match does not exist, for example, empty, or the value of the best match < minimum overlap threshold, then Oi is retained in the recognition result in the collection camera C, since Oi and the best matching item (recorded as O'i), delete any one of them, the other result is a duplicate result, if pi > p'i, it means that the confidence of Oi is high, then Oi is retained, and O'i is deleted from the recognition result of C', if pi ≤ p'i, that is, the confidence of O'i is high, then O'i is retained, and Oi is deleted from the recognition result of C', this step records the correspondence of (C, Oi) and (C', O'i) at the same time, which is used for subsequent steps to calculate the height information of the item, the case where the best match does not exist may be one of the following reasons: one item is not recognized by the adjacent camera, the reason is that the image recognition misses, one item may be blocked due to the angle relationship and not recognized by the adjacent collection camera C, in this case, the item is recognized by the collection camera C, and the multi-collection camera solution perfectly solves this problem; after this step, all duplicate recognized items have been deleted, and the items that are blocked at a certain angle have also been recognized, the total recognition result can be represented as the sum of all camera recognition results, this step describes the multi-camera result fitting algorithm of the invention, which removes the duplicate recognition result and retains the items that are not recognized due to being blocked at a certain angle;

[0016] S7: For the recognized items, judge whether there is stacking: the principle of calculating the distance of the object by binocular vision technology is described as follows: for an actual object point (X, Y, Z), such as (xL, yL) in the left camera image, if its position (xR, yR) in the right camera can be accurately found, then the distance of the object from the camera is: , in the formula, B is the physical distance between the left and right cameras, f is the consistent focal length of the two collection cameras, and it is assumed that yL = yR after binocular correction of the two collection cameras, the difficulty of calculating the distance of the object by binocular vision technology lies in finding the matching points of the left and right (adjacent) cameras, given (xL, yL), the correct (xR, yR) is found by color matching fitting, in step S6, for each item Oi recognized by each camera C, O'i in the adjacent camera C has been found, and the distance of the item is calculated directly, assuming that C and C' are horizontally adjacent and the distance is Dw, the positions of the item in the two cameras are (xi, yi, wi, hi) and (x'i, y'i, w'i, h'i) respectively, then the distance of the center Zc of the item from the camera can be calculated as, , if the distance between the collection camera and the identified region of the object is D, the height of the object is H=D-Zc, if the prior height of the object category ci is Hi, and H is obviously greater than Hi, the object is stacked, and a warning is prompted, C and C' are longitudinally adjacent, and the distance is Dh, so the distance between the center of the object and the collection camera is The height detection method of the application simplifies the description and calculation by using object center point calculation, and the method is more simple and convenient.

[0017] In the application, a large range of recognition is supported, the distance between the camera and the goods can be effectively reduced, the layers of the goods cabinet can be increased, the loading efficiency can be improved, the cost can be reduced, the multi-camera can effectively identify the blocked object due to the camera angle problem, the recognition accuracy can be improved, for the completely stacked scene, the application can effectively prompt the warning, the accurate recognition result can be obtained through manual intervention, and the loss of the merchant can be reduced.

[0018] Preferably, in step S1, the identified region is divided into several sub-regions, and the sub-regions are sequentially labeled as (Q1, Q2,..., Q11, Q12), and the collection cameras are labeled as (C1, C2,..., C5, C6), the distance between each collection camera and the horizontally adjacent collection camera is Dw, and the distance between the longitudinally adjacent collection cameras is Dh.

[0019] Preferably, the frame body 1 is located directly above the identified region, the shooting region of the collection camera C1 is (Q1, Q2, Q5, Q6), the shooting region of the collection camera C2 is (Q2, Q3, Q6, Q7), the shooting region of the collection camera C3 is (Q3, Q4, Q7, Q8), the shooting region of the collection camera C4 is (Q5, Q6, Q9, Q10), the shooting region of the collection camera C5 is (Q6, Q7, Q10, Q11), and the shooting region of the collection camera C6 is (Q7, Q8, Q11, Q12), and each collection camera can shoot an image of an equal region.

[0020] Preferably, in step S2, during the parameter-by-parameter calibration of the acquisition camera, by setting k1, k2, and k3, it is determined whether the imaging of the acquisition camera has radial distortion, by setting p1, p2, and p3, it is determined whether the imaging of the acquisition camera has tangential distortion, and by setting the internal parameters of the camera, i.e., focal length f and imaging origin cx and cy, and five distortion parameters, and the external parameters, i.e., the world coordinates of the calibration object, the relative position between the two cameras, such as the rotation matrix R and the translation vector t of the right camera relative to the left camera, is measured through calibration after the internal parameters of the adjacent cameras are obtained.

[0021] Preferably, in step S2, the color adaptive function (such as white balance and automatic exposure) of the acquisition camera is turned off, and then a gold standard camera is set as a reference, and the RGB colors of all acquisition cameras are adjusted to be consistent with the gold standard camera. The specific steps are as follows:

[0022] S201: Place the standard 24-color color card under the camera to obtain a verification image.

[0023] S202: Process the verification image using a digital image edge detection algorithm to detect the horizontal and vertical division lines of the 24-color color card to obtain a plurality of color blocks of 24 colors.

[0024] S203: Calculate the coordinates of the four corner points of each color block according to the horizontal and vertical division lines of the 24-color color card.

[0025] S204: Calculate the average value of the RGB color of each color block in the image and record it. After this calculation, 24 R values, 24 G values, and 24 B values are obtained.

[0026] S205: Sort the 24 R values obtained by the calibration camera, expressed as (R0, R1, … R23), and for the gold standard camera, there is a known set of R values expressed as (R'0, R'1, … R'23), and the value of R ranges from 0 to 255. Compare (R0, R1, … R23) with (R'0, R'1, … R'23). In this step, the mapping relationship is:

[0027] S206: Map the R value of each pixel of the calibrated camera to R' so that R' approaches the value of the gold standard camera, and finally complete the color calibration of the acquisition camera. Through the fast color calibration method in the present application, the imaging color of the camera is consistent, and the recognition accuracy is improved.

[0028] S207: Repeat steps S205-S206 for the G value and the B value in RGB.

[0029] Preferably, the connecting assembly comprises a first connecting block arranged on the frame body and a second connecting block arranged on the upper end of the collection camera, the first connecting block is connected with the second connecting block through bolts, by arranging the first connecting block and the second connecting block, the collection camera can be replaced after being damaged.

[0030] Preferably, in step S3, for a pair of horizontally adjacent cameras, binocular correction is performed according to monocular internal parameters (focal length, imaging origin, distortion coefficient) obtained after camera calibration and binocular relative position relationship (rotation matrix and translation vector), respectively, to eliminate distortion and line alignment of left and right views, so that the imaging origin coordinates of left and right views are consistent, the optical axes of two cameras are parallel, the left and right imaging planes are coplanar, and the epipolar lines are aligned, and after correction, any point on one image and its corresponding point on another image necessarily have the same line number. In binocular matching, only the same signal of adjacent cameras needs to be searched, and similarly, for a pair of longitudinally adjacent cameras, after binocular correction, any point on one image and its corresponding point on another image necessarily have the same column number.

[0031] Preferably, in step S5, the confidence threshold is whether the ratio of the prediction box and the labeled box is greater than a set threshold, so as to determine whether the prediction result of the detection model is a positive sample. Generally, the threshold is set to 0.5, but it can also be 0.1, 0.25 or 0.7. By setting different thresholds, many groups of precision and recall rates are obtained. Among multiple detection targets, the precision rate is taken as the vertical coordinate and the recall rate is taken as the horizontal coordinate, and a PR curve can be drawn for each detection category. The evaluation index AP is the area under the PR curve, and mAP is the average of all detection category AP values. mAP is the most important evaluation index of the target detection model, which measures the detection effect of the trained detection model on all categories. The larger the value is, the better the trained model is.

[0032] Compared with the prior art, the present application has the following beneficial effects:

[0033] This invention is particularly significant for scenarios such as point-of-sale (POS) and smart vending machines. It supports wide-range recognition, effectively reducing the distance between the camera and the goods, thereby increasing the number of shelves in the vending machine, improving loading efficiency, and reducing costs. For obstructions caused by camera angles, multiple cameras can effectively identify obstructed objects, improving recognition accuracy. In scenarios with completely stacked items, this invention can effectively provide warnings, allowing for accurate recognition results after manual intervention, reducing losses for merchants. This invention offers excellent recognition performance, accurately identifying both obstructed and stacked items. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the process structure of a method for large-area object recognition using multiple cameras according to the present invention;

[0035] Figure 2 This is a flowchart of step S2 in a method for large-area object recognition using multiple cameras according to the present invention;

[0036] Figure 3 This is a schematic diagram of the overall structure of the frame, training components, acquisition camera, and recognition area in this invention;

[0037] Figure 4 This is a schematic diagram of the main structure of the frame, training components, acquisition camera, and recognition area in this invention;

[0038] Figure 5 For the present invention Figure 4 A magnified schematic diagram of the local structure at point A;

[0039] Figure 6 This is a code diagram of step S6 in the present invention;

[0040] Figure 7 A schematic diagram of the first existing technology structure;

[0041] Figure 8 The second existing technology structure diagram.

[0042] In the diagram: 1. Frame; 2. Connecting components; 21. First connecting block; 22. Second connecting block. Detailed Implementation

[0043] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments.

[0044] In the description of the present application, it should be noted that the terms "upper", "lower", "inner", "outer", "front end", "rear end", "two ends", "one end", "the other end" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application. In addition, the terms "first", "second" are only for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0045] In the description of the present application, it should be noted that unless otherwise explicitly specified and limited, the terms "mounting", "provided with", "connection" and the like should be understood in a broad sense, for example, "connection" can be fixed connection, can also be detachable connection, or integral connection; can be mechanical connection, can also be electrical connection; can be directly connected, can also be indirectly connected through an intermediate medium, can be internal communication of two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0046] Please refer to Figures 1-8 The present application is a method for identifying a wide range of goods using multiple cameras, comprising the following steps:

[0047] S1: setting a frame body 1 on the identified area, the lower end of the frame body 1 is provided with a connecting component 2, a plurality of connecting components 2 are distributed at equal intervals, and the lower end of the connecting component 2 is provided with a plurality of collection cameras, this step ensures that each identification area is photographed by at least two adjacent cameras;

[0048] S2: first calibrating the collection cameras by parameters, and then calibrating the colors,

[0049] S3: calibrating each pair of adjacent collection cameras;

[0050] S4: identifying goods for each collection camera image; inputting the image of each collection camera into an artificial intelligence image recognition network on a computer to obtain all the recognized objects in the image, for the recognized object i, it is expressed as Oi = (ci, pi, xi, yi, wi, hi), wherein: ci represents the object category, pi represents the recognition confidence, xi and yi are the coordinates of the upper left corner of the object rectangular frame, and wi and hi are the width and height of the object rectangular frame;

[0051] S5: for the identification result of each collection camera, filter out the goods with low confidence, if the collection camera C recognizes the object Oi, and pi < confidence threshold, then delete this object from the identification result, so that for each collection camera C, an identification result set ;

[0052] S6: For each pair of adjacent cameras, the overlapping area is fitted, and the specific steps of fitting are as follows:

[0053] S601: For any object Oi recognized by the collection camera C, calculate the area A where Oi is located, and the corresponding adjacent collection camera C imaging area is divided into 4 parts: upper left A0, lower left A1, upper right A2 and lower right A3, each area and the front and rear adjacent collection camera overlap, calculate the overlap coefficient γ of Oi and the area Ak (k=0, 1, 2, 3) The formula is: , and then the formula for calculating the area where Oi is located is, From the above formula, if the overlap coefficient of Oi and Ak is the largest, it is considered that Oi belongs to this area Ak;

[0054] S602: When Oi belongs to the area A, for each camera C' adjacent to A, obtain all the recognized objects of class ci in the area A, denoted as ', and the judgment of whether the recognition result of C' belongs to A is calculated by the formula in step S601, and then the objects of class ci recognized by the adjacent collection camera in the area A are found. This step is mainly used to confirm whether it is repeated recognition in the next step;

[0055] S603: Calculate the best match of Oi and ' all recognized objects, for the recognition result Oj in ', the matching calculation between Oi and Oj is: , and the best matching object of Oi and φ' is: , if the best match does not exist, for example, empty, or the value of the best match < minimum overlap threshold, then Oi is retained in the recognition result in the acquisition camera C, since Oi and the best matching item (recorded as O'i), delete any one of them, the other result is a duplicate result, if pi > p'i, it means that the confidence of Oi is high, then Oi is retained, and O'i is deleted from the recognition result of C', if pi < p'i, that is, the confidence of O'i is high, then O'i is retained, and Oi is deleted from the recognition result of C', this step records the correspondence of (C, Oi) and (C', O'i) at the same time, which is used for subsequent steps to calculate the height information of the item, the case where the best match does not exist may be one of the following reasons: one item is not recognized by the adjacent camera, the reason is that the image recognition misses, one item may be blocked due to the angle relationship and not recognized by the adjacent acquisition camera C, in this case, the item is recognized by the acquisition camera C, and the multi-acquisition camera scheme perfectly solves this problem; after this step, all duplicate recognized items have been deleted, and the items that are blocked at a certain angle have also been recognized, the total recognition result can be represented as the sum of all camera recognition results, this step describes the multi-camera result fitting algorithm of the invention, which removes the duplicate recognition results while retaining the items that are not recognized due to being blocked at a certain angle;

[0056] S7: For the recognized items, determine whether there is a stack: the principle of calculating the distance of the object by binocular vision technology is described as follows: for an actual object point (X, Y, Z), such as (xL, yL) in the left camera image, if its position (xR, yR) in the right camera can be accurately found, then the distance of the object from the camera is: , in the formula, B is the physical distance between the left and right cameras, f is the consistent focal length of the two acquisition cameras, and it is assumed that yL = yR after binocular correction of the two acquisition cameras, the difficulty of calculating the distance of the object by binocular vision technology lies in finding the matching points of the left and right (adjacent) cameras, given (xL, yL), the correct (xR, yR) is found by color matching fitting, in step S6, for each item Oi recognized by each camera C, O'i in the adjacent camera C has been found, and the distance of the item is calculated directly, assuming that C and C' are horizontally adjacent and the distance is Dw, the positions of the item in the two cameras are (xi, yi, wi, hi) and (x'i, y'i, w'i, h'i) respectively, then the distance of the center Zc of the item from the camera can be calculated as, , if the distance between the collection camera and the identified region of the object is D, the height of the object is H=D-Zc, if the prior height of the object category ci is Hi, and H is obviously greater than Hi, the object is stacked, and a warning is prompted, C and C' are longitudinally adjacent, and the distance is Dh, so the distance between the center of the object and the collection camera is The height detection method of the application simplifies the description and calculation by using object center point calculation, and the method is more simple and convenient.

[0057] In the application, a large range of recognition is supported, the distance between the camera and the goods can be effectively reduced, the layers of the goods cabinet can be increased, the loading efficiency can be improved, the cost can be reduced, the multi-camera can effectively recognize the blocked object due to the camera angle problem, the recognition accuracy can be improved, for the completely stacked scene, the application can effectively prompt the warning, the accurate recognition result can be obtained through manual intervention, and the loss of the merchant can be reduced.

[0058] In step S1, the identified region is divided into several sub-regions, and the sub-regions are sequentially labeled as (Q1, Q2,..., Q11, Q12), and the collection cameras are labeled as (C1, C2,..., C5, C6), the distance between each collection camera and the horizontally adjacent collection camera is Dw, and the distance between the longitudinally adjacent collection cameras is Dh.

[0059] In step S1, the identified region is divided into several sub-regions, and the sub-regions are sequentially labeled as (Q1, Q2,..., Q11, Q12), and the collection cameras are labeled as (C1, C2,..., C5, C6), the distance between each collection camera and the horizontally adjacent collection camera is Dw, and the distance between the longitudinally adjacent collection cameras is Dh.

[0060] In step S2, during the parameter calibration of the acquisition camera, three parameters k1, k2 and k3 are set to determine whether radial distortion exists in the imaging of the acquisition camera, three parameters p1, p2 and p3 are set to determine whether tangential distortion exists in the imaging of the acquisition camera, and five distortion parameters of the internal parameters of the camera, i.e., focal length f and imaging origin cx and cy, and external parameters, i.e., world coordinates of the calibration object, are set.

[0061] In step S2, the color adaptive function (for example, white balance and automatic exposure) of the acquisition camera is turned off, and a gold standard camera is set as a reference, and the RGB colors of all acquisition cameras are adjusted to be consistent with the gold standard camera, and the specific steps are as follows:

[0062] S201: Place the standard 24-color color card under the camera to obtain a verification image.

[0063] S202: Process the verification image by using a digital image edge detection algorithm to detect the horizontal and vertical division lines of the 24-color color card to obtain a plurality of color blocks of 24 colors.

[0064] S203: Calculate the coordinates of the four corner points of each color block according to the horizontal and vertical division lines of the 24-color color card.

[0065] S204: Calculate the average value of the RGB color of each color block in the image, and record the values. After the calculation, 24 R values, 24 G values and 24 B values are obtained.

[0066] S205: Sort the 24 R values obtained by the calibration camera, which are expressed as (R0, R1, … R23), and for the gold standard camera, there is a known R value expressed as (R'0, R'1, … R'23), and the value of R ranges from 0 to 255. Compare (R0, R1, … R23) with (R'0, R'1, … R'23). In this step, the mapping relationship is:

[0067] S206: Map the R value of each pixel of the calibrated camera to R' so that R' approaches the value of the gold standard camera, and finally complete the color calibration of the acquisition camera. In the present application, the color calibration method ensures that the colors of the camera imaging are consistent, and the recognition accuracy is improved.

[0068] S207: Repeat steps S205-S206 for the G value and B value in RGB.

[0069] The connecting assembly 2 comprises a first connecting block 21 arranged on the frame body 1 and a second connecting block 22 arranged at the upper end of the collection camera, the first connecting block 21 is connected with the second connecting block 22 through bolts, and the first connecting block 21 and the second connecting block 22 are arranged, so that the collection camera can be replaced after being damaged.

[0070] In step S3, for a pair of horizontally adjacent cameras, binocular correction is performed according to monocular internal parameter data (focal length, imaging origin, distortion coefficient) obtained after camera calibration and binocular relative position relationship (rotation matrix and translation vector), respectively, to eliminate distortion and line alignment for left and right views, so that the imaging origin coordinates of the left and right views are consistent, the optical axes of the two cameras are parallel, the left and right imaging planes are coplanar, and the epipolar lines are aligned, and after correction, any point on one image and its corresponding point on another image necessarily have the same line number. In binocular matching, the matching point only needs to search for the same signal of the adjacent camera, and similarly, for a pair of longitudinally adjacent cameras, after binocular correction, any point on one image and its corresponding point on another image necessarily have the same column number.

[0071] In step S5, the confidence threshold is whether the ratio of the prediction box to the labeled box is greater than a set threshold, so as to determine whether the prediction result of the detection model is a positive sample, generally, the threshold is set to 0.5, which can also be 0.1, 0.25 or 0.7, by setting different thresholds, a lot of precision and recall rate is obtained, in multiple detection targets, taking the precision rate as the vertical coordinate and the recall rate as the horizontal coordinate, a PR curve can be drawn for each detection category, the evaluation index AP is the area under the PR curve, and mAP is the average of all detection category AP values, mAP is the most important evaluation index of the target detection model, which measures the detection effect of the trained detection model on all categories, and the larger the value is, the better the trained model is.

[0072] The basic principles and main features of the present application are shown and described above, and the advantages of the present application are shown and described above. Those skilled in the art should understand that the present application is not limited to the above examples, the above examples and descriptions in the specification are only to illustrate the principles of the present application, and various changes and improvements can be made without departing from the spirit and scope of the present application, and these changes and improvements all fall within the scope of the claimed present application. The scope of protection of the present application is defined by the appended claims and their equivalents.

Claims

1. A method for large-scale item recognition using multiple cameras, the method comprising: The method comprises the following steps: ​ S1: setting a frame body (1) on the identified area, the lower end of the frame body (1) is provided with a plurality of connection assemblies (2), the connection assemblies (2) are arranged at equal intervals, and the lower end of the connection assembly (2) is provided with a plurality of collection cameras; S2: performing parameter calibration on the collection camera first, and then performing color calibration; S3: performing binocular correction on each pair of adjacent collection cameras; S4: performing object identification on each collection camera image; inputting the image of each collection camera into an artificial intelligence image recognition network on a computer to obtain all recognized objects in the image, and the recognized object i is expressed as Oi=(ci, pi, xi, yi, wi, hi), wherein: ci represents the object category, pi represents the recognition confidence, xi and yi are the coordinates of the upper left corner of the object rectangular frame, and wi and hi are the width and height of the object rectangular frame; S5: filtering out low-confidence objects for each collection camera recognition result, if the collection camera C recognizes an object Oi, and pi is less than the confidence threshold, the object is deleted from the recognition result, so that a recognition result set φ is obtained for each collection camera C; S6: fitting the results of each pair of adjacent cameras in the overlapping area, and the specific steps are as follows: S601: For any object Oi identified by the collection camera C, calculate the region A where Oi is located, and the corresponding adjacent imaging region of the collection camera C is divided into 4 parts: upper left A0, lower left A1, upper right A2 and lower right A3, each region overlaps the front and rear and left and right adjacent collection camera, and the overlap coefficient γ of Oi and the region Ak is calculated according to the formula: , and then the formula for calculating the region where Oi is located is, According to the above formula, if the overlap coefficient of Oi and Ak is the largest, it is considered that Oi belongs to this region Ak. S602: When Oi belongs to region A, for each camera C' adjacent to A, obtain all the identified items of category ci within region A, using C' to indicate whether the identification result of C' belongs to A, the formula in step S601 is calculated, and then the items of category ci that are also identified within region A by the adjacent collection camera are found; S603: Calculate Oi and 'The best match for all identified items within, for The matching calculation between the recognition result Oj and Oi within the range is as follows: Calculate Oi and The best matching item is: If the best match does not exist, or the value of the best match is less than the minimum overlap threshold, then Oi is retained in the recognition result of the acquisition camera C. Since Oi and the best matching item are recorded in O'i, if pi > p'i, it means that the confidence of Oi is high, so Oi is retained and O'i is deleted from the recognition result of C'. If pi ≤ p'i, that is, the confidence of O'i is high, then O'i is retained and Oi is deleted from the recognition result of C'. S7: judging whether there is stacking for the recognized object: the principle of calculating the distance of the object by binocular vision technology is described as follows: for a point (X, Y, Z) of the actual object, such as the position (xL, yL) in the left camera image, if the position (xR, yR) in the right camera can be accurately found, the distance of the object from the camera is: In the formula, B is the physical distance between the left and right cameras, f is the consistent focal length of the two collection cameras, and yL=yR is set after binocular correction of the two collection cameras. The difficulty of calculating the distance of the object by binocular vision technology lies in finding the matching points of the left and right cameras. Given (xL, yL), the correct (xR, yR) is found by color matching fitting. In step S6, for each object Oi recognized by each camera C, O’i in the adjacent camera C’ has been found, and the distance of the object is directly calculated. Assuming that C and C’ are horizontally adjacent and the distance is Dw, the positions of the object in the two cameras are (xi, yi, wi, hi) and (x’i, y’i, w’i, h’i), respectively, and the distance of the center Zc of the object from the camera can be calculated as, Assuming that the distance between the collection camera and the recognized area of the object is D, the height of the object is H=D-Zc. If the prior height of the object category ci is Hi, and H is obviously greater than Hi, it can be concluded that the object is stacked, and a warning is prompted. Assuming that C and C’ are vertically adjacent and the distance is Dh, the distance of the center of the object from the collection camera is . 2.The method of claim 1, wherein: In step S1, the identified area is divided into a plurality of sub-regions, and the sub-regions are sequentially labeled as (Q1, Q2,..., Q11, Q12). A plurality of collection cameras are labeled as (C1, C2,..., C5, C6). The distance between each collection camera and the horizontally adjacent collection camera is Dw, and the distance between each collection camera and the vertically adjacent collection camera is Dh. 3.The method of claim 2, wherein: The frame body (1) is located directly above the identified area, the shooting area of the collection camera C1 is Q1, Q2, Q5 and Q6; the shooting area of the collection camera C2 is Q2, Q3, Q6 and Q7; the shooting area of the collection camera C3 is Q3, Q4, Q7 and Q8; the shooting area of the collection camera C4 is Q5, Q6, Q9 and Q10; the shooting area of the collection camera C5 is Q6, Q7, Q10 and Q11; and the shooting area of the collection camera C6 is Q7, Q8, Q11 and Q12.

4. The method of claim 3, wherein: In step S2, during the parameter calibration of the collection camera, three parameters k1, k2 and k3 are set to determine whether the collection camera imaging has radial distortion, three parameters p1, p2 and p3 are set to determine whether the collection camera imaging has tangential distortion, and the internal parameters of the camera, such as focal length f and imaging origin cx and cy, and five distortion parameters are set; and the external parameters of the calibration object, such as the world coordinates.

5. The method of claim 4, wherein: In step S2, the color adaptive function of the collection camera is confirmed to be turned off, then a gold standard camera is set as a reference, and the RGB colors of all collection cameras are adjusted to be consistent with the gold standard camera, and the specific steps are as follows: S201: Place the standard 24-color color card under the camera to obtain a verification image; S202: Process the verification image using a digital image edge detection algorithm to detect the horizontal and vertical division lines of the 24-color color card, and obtain a plurality of color blocks of 24 colors; S203: Calculate the coordinates of the four corner points of each color block according to the horizontal and vertical division lines of the 24-color color card; S204: Calculate the average value of the RGB color of each color block in the image, and record the 24 R values, 24 G values, and 24 B values obtained after the calculation; S205: Sort the 24 R values obtained by the calibrated camera, expressed as (R0, R1, … R23), and for the gold standard camera, there is a known R value expressed as (R'0, R'1, … R'23), and the value of R is in the range of [0, 255], compare (R0, R1, … R23) with (R'0, R'1, … R'23); S206: Map the R value of each pixel of the calibrated camera to R', so that R' approaches the value of the standard camera, and finally complete the color calibration of the acquisition camera; S207: Repeat steps S205-S206 for G and B values in RGB. 6.The method of claim 1, wherein: The connection assembly 2 includes a first connecting block (21) arranged on the frame body 1 and a second connecting block (22) arranged at the upper end of the acquisition camera, and the first connecting block (21) and the second connecting block (22) are connected by bolts.

7. The method of claim 1, wherein: In step S3, for a pair of horizontally adjacent cameras, binocular correction is performed according to the monocular internal parameter data obtained after camera calibration and the relative position relationship of binoculars, respectively, to eliminate distortion and line alignment of left and right views, so that the imaging origin coordinates of left and right views are consistent, the optical axes of two cameras are parallel, the left and right imaging planes are coplanar, and the epipolar lines are aligned. 8.The method of claim 1, wherein: In step S5, the confidence threshold is whether the ratio of the prediction box to the labeled box is greater than a set threshold, so as to determine whether the prediction result of the detection model is a positive sample.

Images