Chassis misloading detection system, method, and apparatus

Abstract

The application provides a chassis missing loading detection system, method and device, and belongs to the technical field of vehicle manufacturing. The method comprises the following steps: acquiring chassis initial data, and determining detection configuration parameters according to the chassis initial data; controlling at least two video stream acquisition devices and multiple light sources to start according to the detection configuration parameters, so as to obtain chassis video streams of multiple regions of the chassis; determining missing loading information of parts corresponding to each region according to the chassis video stream of each region, pre-stored standard part information and a pre-constructed model; and performing chassis splicing and detection result fusion processing according to all the missing loading information of the parts, all the chassis video streams and the detection configuration parameters, so as to obtain a target detection result. The application has the advantages of improving detection efficiency and stability of detection results.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of vehicle manufacturing, in particular to a chassis missing detection system, method and device. BACKGROUND

[0002] With the gradual increase in demand for vehicles, production processes that can improve the production efficiency of vehicles have gradually been valued by the industry. Among them, the efficiency and accuracy of chassis missing detection in the entire vehicle production process play an important role in the vehicle production cycle.

[0003] At present, in the prior art, the chassis missing detection link in the vehicle production process mainly includes visual observation of whether the buckle, bolt and screw at the chassis card plate installation place are missing by manual observation. The workers at the detection station need to walk back and forth to observe the entire chassis area of 5 to 6 meters long and about 2 meters wide to complete the missing detection.

[0004] However, the inventors have found that the prior art at least has the following technical problems: since the chassis missing detection link is mainly completed by manual observation, and the observation results are easily affected by subjective factors and fatigue of the person. This leads to low detection efficiency and poor detection result stability. SUMMARY

[0005] The present application provides a chassis missing detection system, method and device to improve detection efficiency and detection result stability.

[0006] In a first aspect, the present application provides a chassis missing detection system, comprising: at least two video stream acquisition devices, a plurality of light sources and a controller;

[0007] The at least two video stream acquisition devices are arranged side by side at intervals, and the shooting direction of each video stream acquisition device is towards the chassis.

[0008] The plurality of light sources are arranged in a rectangular shape around the at least two video stream acquisition devices; and the controller is in communication connection with the at least two video stream acquisition devices.

[0009] In a possible implementation, the number of video stream acquisition devices is two, the two video stream acquisition devices are cross-calibrated and aligned in the horizontal direction and the vertical direction, the horizontal field of view of the two video stream acquisition devices covers the width of the chassis; the light source is a strip light, the number of strip lights is 10, and the angle adjustment range of each strip light is 0 to 360 degrees.

[0010] In a second aspect, the present application provides a chassis missing detection method applied to the controller in the chassis missing detection system described in the first aspect, wherein the chassis missing detection system comprises: at least two video stream acquisition devices, a plurality of light sources and a controller;

[0011] The chassis missing detection method comprises:

[0012] Obtaining chassis initial data, and determining detection configuration parameters according to the chassis initial data;

[0013] Controlling the at least two video stream acquisition devices and the plurality of light sources to start according to the detection configuration parameters to obtain chassis video streams of a plurality of regions of the chassis;

[0014] Determining missing part information of each region corresponding to each region according to the chassis video stream of each region, pre-stored standard part information and a pre-constructed model;

[0015] Performing chassis splicing and detection result fusion processing according to all the missing part information, all the chassis video streams and the detection configuration parameters to obtain a target detection result.

[0016] In a possible implementation, the detection configuration parameters comprise a shooting exposure time corresponding to each region of the chassis. Correspondingly, the controlling the at least two video stream acquisition devices and the plurality of light sources to start according to the detection configuration parameters to obtain chassis video streams of a plurality of regions of the chassis comprises: when it is detected that one region of the chassis enters a shooting range of the at least two video stream acquisition devices, controlling the at least two video stream acquisition devices and the plurality of light sources to acquire a first initial chassis video stream and a second initial chassis video stream of the region according to the shooting exposure time corresponding to the region; determining a first region of interest (ROI) picture corresponding to the region according to the first initial chassis video stream of the region, and determining a second ROI picture corresponding to the region according to the second initial chassis video stream of the region; and determining the first ROI picture and the second ROI picture as the chassis video stream of the region.

[0017] In a possible implementation, the determining missing part information of each region corresponding to each region according to the chassis video stream of each region, pre-stored standard part information and a pre-constructed model comprises: determining a preliminary part detection result of the region according to the chassis video stream of each region and the pre-constructed model; determining an Euclidean distance result value between each part in the region and a corresponding pre-stored standard part according to the preliminary part detection result of the region and the pre-stored standard part information, and when it is detected that the Euclidean distance result value is greater than a preset error threshold value, determining that the pre-stored standard part is a missing part, and determining a missing position and a missing type according to the pre-stored standard part information; determining the missing position of all the missing parts as the missing part position of the region, and determining the missing type of all the missing parts as the missing part type of the region.

[0018] In a possible implementation, the determining of the missing part information corresponding to each region according to the chassis video stream of each region, the pre-stored standard part information, and the pre-constructed model comprises: determining a preliminary part detection result of the region according to the chassis video stream of each region and the pre-constructed model; determining a missing part position of the region according to the preliminary part detection result of the region; determining a Euclidean distance result value between each missing part and each pre-stored standard part according to the pre-stored standard part information and the missing part position of the region, and determining a pre-stored standard part corresponding to a minimum Euclidean distance result value as a missing part type of the missing part; and determining the missing part position and the missing part type of the region according to the pre-stored standard part information and the missing part types of all missing parts.

[0019] In a possible implementation, the detection configuration parameter comprises a preset picture overlap ratio. Correspondingly, the chassis splicing and detection result fusion processing according to all the missing part information, all the chassis video streams, and the detection configuration parameter to obtain a target detection result comprises: performing height correction processing on the first ROI picture and the second ROI picture of each region to obtain a first to-be-spliced ROI picture and a second to-be-spliced ROI picture that are consistent in height; determining a width value and a height value of a reference picture according to the first to-be-spliced ROI picture and the second to-be-spliced ROI picture of all regions; performing splicing processing on the first to-be-spliced ROI picture of all regions according to the height value of the reference picture in a first alignment manner to obtain a first spliced picture, and performing splicing processing on the second to-be-spliced ROI picture of all regions according to the height value of the reference picture in a second alignment manner to obtain a second spliced picture; performing splicing processing on the first spliced picture and the second spliced picture according to the width value of the reference picture in a chassis width direction according to a preset picture overlap ratio to obtain a target spliced picture comprising a plurality of regions of the chassis; and performing detection result information fusion processing on the target spliced picture and all the missing part information to obtain a target detection result.

[0020] In a possible implementation, the height correction processing on the first ROI picture and the second ROI picture of each region to obtain a first to-be-spliced ROI picture and a second to-be-spliced ROI picture that are consistent in height comprises: performing background brightness filling processing on the first ROI picture and the second ROI picture of each region according to a maximum height value of the region to obtain the first to-be-spliced ROI picture and the second to-be-spliced ROI picture that are consistent in height.

[0021] In a possible implementation, the determining the width value and the height value of the reference picture according to the first ROI picture and the second ROI picture of all regions comprises: determining a region width value according to the first ROI picture and the second ROI picture of each region, performing comparison processing on all region width values, and determining a maximum value in the all region width values as the width value of the reference picture; determining a first ROI picture height accumulation value according to the first ROI picture of all regions, and determining a second ROI picture height accumulation value according to the second ROI picture of all regions; when it is detected that the first ROI picture height accumulation value is greater than the second ROI picture height accumulation value, determining the first ROI picture height accumulation value as the height value of the reference picture.

[0022] In a third aspect, the application provides a chassis missing detection device, applied to a controller in the chassis missing detection system as described in the first aspect, wherein the chassis missing detection system comprises: at least two video stream acquisition devices, a plurality of light sources, and the controller.

[0023] The chassis missing detection device comprises:

[0024] The acquisition module is configured to acquire chassis initial data, and determine detection configuration parameters according to the chassis initial data.

[0025] The acquisition module is further configured to control the at least two video stream acquisition devices and the plurality of light sources to start according to the detection configuration parameters, and obtain chassis video streams of a plurality of regions of the chassis.

[0026] The detection module is configured to determine part missing information corresponding to each region according to the chassis video stream of each region, pre-stored standard part information, and a pre-constructed model.

[0027] The splicing and fusion module is configured to perform chassis splicing and fusion processing according to all part missing information, all chassis video streams, and the detection configuration parameters, and obtain a target detection result.

[0028] This application provides a chassis missing parts detection system, method, and apparatus. After acquiring initial chassis data, detection configuration parameters are determined based on this data. Then, according to the detection configuration parameters, a light source is controlled to illuminate the chassis, and at least two video stream acquisition devices are simultaneously controlled to acquire video streams from each area of ​​the moving chassis, resulting in multiple chassis video streams. A preliminary inspection of the chassis is then performed based on the chassis video streams from each area, pre-stored standard part information, and a pre-built model to determine the missing parts information for each area. Finally, based on all the missing parts information, all the chassis video streams, and the detection configuration parameters, chassis splicing and detection result fusion processing are performed to obtain the target detection result. The entire detection process is less affected by human factors, and the detection of chassis missing parts can be automatically completed and the detection results obtained during chassis movement, improving detection efficiency and the stability of the detection results. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a schematic diagram of the chassis missing parts detection system provided in the embodiments of this application;

[0031] Figure 2 A schematic flowchart illustrating the chassis missing parts detection method provided in this application embodiment;

[0032] Figure 3 A schematic diagram of the target stitched image provided in the embodiments of this application;

[0033] Figure 4 This is a schematic diagram of the chassis missing parts detection device provided in the embodiments of this application;

[0034] Figure 5 A schematic diagram of the hardware structure of the chassis missing parts detection device provided in this application embodiment.

[0035] Figure label:

[0036] 101-Video stream acquisition device; 102-Light source; 103-Controller; 41-Acquisition module; 42-Detection module; 43-Stabbing and fusion module; 501-Processor; 502-Memory; 503-Bus. Detailed Implementation

[0037] To make the purposes, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0038] At present, the types of parts required to be detected whether missing on the vehicle chassis are various, the quantity is large, and the area to be detected is large. The manual detection station needs to walk back and forth to observe, the detection efficiency is low, and with the increase of working time, visual fatigue is easy to occur, resulting in missed detection of missing points in the detection process, and there are problems of low detection efficiency and poor stability of detection results.

[0039] To solve the above technical problems, the embodiments of the present application provide the following technical ideas to solve the problems: first, using left and right binocular cameras and rectangularly arranged strip tubes, video streams of each different area of the vehicle chassis moving at a constant speed are collected, and based on a deep learning model, visual interest point ROI pictures of multiple areas are detected for missing parts, to improve the detection efficiency and the stability of the detection results.

[0040] Figure 1 The structure diagram of the chassis missing detection system provided by the embodiments of the present application is shown in FIG. 1, which includes at least two video stream collection devices 101, a plurality of light sources 102, and a controller 103. Figure 1

[0041] The at least two video stream collection devices 101 can be a camera, a video recorder, or a camera that can shoot videos, and are fixed side by side on the support placed under the chassis in advance through bolt connection, so that the shooting direction of each video stream collection device is toward the chassis, to complete the collection of the video stream of the chassis.

[0042] In an optional embodiment of the present application, the number of video stream collection devices 101 is two, and the two video stream collection devices 101 can be selected as cameras with a horizontal field of view of about half of the width of the chassis or the width of the chassis. The horizontal field of view of the two cameras can cover the entire width of the chassis, and the collection of the video stream of the vehicle chassis is completed. Cross calibration is required when the two cameras are installed to ensure that the horizontal direction and the vertical direction of the two cameras are aligned.

[0043] The light source 102 can adopt a strip light, and the number of light sources 102 can be set according to the width of the chassis. The plurality of light sources 102 are arranged in a rectangle around the two video stream collection devices 101.

[0044] ​If the chassis width is two meters, in an optional embodiment of the present application, the number of strip lights can be set to 10, and a rectangular arrangement is formed around the two video stream acquisition devices 101, such as two cameras, with two strip lights on each of the two opposite sides, and three strip lights on each of the other two opposite sides. Moreover, in order to adapt to different types of chassis, the angle of each strip light is adjusted in the range of 0 to 360 degrees. In this way, the light reflected by various parts such as bolts, screws, and buckles on the chassis can be received by the eight neighborhoods of the two cameras, so that the bolts, screws, and buckles are clearly presented.

[0045] The controller 103 can be a device, a cloud server, or a related device of other computers with data processing function, control function, and missing detection function. The controller 103 is in communication connection with the at least two video stream acquisition devices 101. The controller 103 is configured to receive the video stream acquired by the at least two video stream acquisition devices 101, and use the video stream to complete the entire process of chassis missing detection, and output the detection result.

[0046] The following will be described in conjunction with the accompanying drawings Figure 2 The chassis missing detection method will be described.

[0047] Figure 2 The flowchart of the chassis missing detection method provided by the embodiment of the present application. The execution subject of the embodiment can be Figure 1 The controller 103 in the chassis missing detection system embodiment shown in the figure can also be a related device of other computers. The embodiment is not particularly limited, and the chassis missing detection system includes at least two video stream acquisition devices 101, a plurality of light sources 102, and a controller 103. The connection relationship and purpose of each component in the chassis missing detection system have been described in the embodiment shown in the figure, so the embodiment will not be described here. Figure 1

[0048] As shown in the figure, the chassis missing detection method includes Figure 2

[0049] S201: Obtain chassis initial data, and determine detection configuration parameters according to the chassis initial data.

[0050] In the embodiment, the chassis initial data can be the type of the chassis, the corresponding color of each card plate on the chassis of the type, or the part type information corresponding to the card plate to be detected. The detection configuration parameters can be corresponding system configuration parameters loaded for different types of chassis, for example, splicing and fusion parameters corresponding to different chassis types.

[0051] S202: Control the at least two video stream acquisition devices 101 and the plurality of light sources 102 to start according to the detection configuration parameters, and obtain chassis video streams of a plurality of regions of the chassis.​​

[0052] In this embodiment, the chassis video stream of each region of the chassis refers to that each chassis can be divided into multiple regions, and the chassis video stream of each region is sequentially captured until the entire chassis is captured when the chassis is captured by using two video stream acquisition devices 101.

[0053] In an optional embodiment of the present application, the detection configuration parameter can include a shooting exposure time corresponding to each region of the chassis; accordingly, the at least two video stream acquisition devices 101 and the multiple light sources 102 are controlled to start according to the detection configuration parameter, to obtain the chassis video stream of multiple regions of the chassis, including:

[0054] S202a: when it is detected that a region of the chassis enters the shooting range of the at least two video stream acquisition devices 101, the at least two video stream acquisition devices 101 and the multiple light sources 102 are controlled to capture a first initial chassis video stream and a second initial chassis video stream of the region according to the shooting exposure time corresponding to the region.

[0055] In this embodiment, the chassis can be divided into multiple regions, for example, 6 regions, from the front to the tail of the chassis according to the type of the card board material, and the controller 103 can set an interval of the top position of the 6 regions as a judgment basis for the arrival position of the 6 regions in the video stream according to the chassis picture sample captured by the video stream acquisition device 101. When it is detected that a region of the chassis enters the shooting range, the at least two video stream acquisition devices 101 are controlled to shoot, and the camera exposure time corresponding to the region entering the shooting range is adjusted during shooting. The camera exposure time can be an exposure time less than 10000 microseconds, such as 0.05 seconds or 0.02 seconds. In this way, the blurred chassis pictures during shooting can be prevented. In addition, the lighting time needs to be controlled, that is, the time length from when the region enters the shooting range to when the region leaves the shooting range is the lighting time of the light source 102 for the region. When the system detects the last region of the chassis or the tail of the chassis, all the light sources 102 and the video stream acquisition devices 101 are turned off, and the video stream acquisition work is restarted until the front of the next chassis enters the shooting range of the video stream acquisition device 101.

[0056] The first initial video stream refers to the video stream shot by the first row of video acquisition devices for the region, and the second initial video stream refers to the video stream shot by the second row of video acquisition devices for the region. Until the acquisition of the entire chassis video stream is completed. During this process, the position of each frame of picture of the video stream acquisition device 101 needs to be judged, and when it is detected that a certain region reaches a pre-set boundary position, the initial video stream of the video stream acquisition device 101 shooting the region at this moment is saved.

[0057] S202b: determining a first region of interest (ROI) picture corresponding to the region according to the first initial chassis video stream of the region, and determining a second ROI picture corresponding to the region according to the second initial chassis video stream of the region.

[0058] S202c: determining the first ROI picture and the second ROI picture as the chassis video stream of the region.

[0059] In the embodiment, the region of interest (ROI) picture refers to a region that needs to be further processed and analyzed extracted from the original picture in the initial video stream collected by the video stream collection device 101. For example, in the embodiment, the ROI picture refers to a region on the chassis that needs to be detected for missing parts. When the last region of the chassis is detected or the tail of the chassis is detected, the detection of the region position of the entire chassis is ended, and at this time, the ROI pictures of multiple regions of the video stream collection device 101 are completed.

[0060] In an optional embodiment of the present application, when the number of video stream collection devices 101 is two, for the convenience of description, the following video stream collection devices 101 are two, and are referred to as a left camera and a right camera. And the chassis is divided into 12 regions from the head to the tail according to the material of the card plate, therefore, the left camera collects first ROI pictures of 6 regions, and the right camera collects second ROI pictures of 6 regions. The first ROI picture and the second ROI picture in the same width direction can be determined as the chassis video stream of the region in the width direction.

[0061] S203: determining part missing information corresponding to each region according to the chassis video stream of each region, pre-stored standard part information, and a pre-constructed model.

[0062] In the embodiment, the pre-stored standard part information can include installation position information of a standard part and a type of the standard part. The pre-constructed model is a prediction model that can be continuously learned and updated using a training set in an offline model training process, and has more accurate detection results. An algorithm model with region detection, part detection, and missing detection functions is obtained. The part missing information corresponding to each region can include a type and a position of a missing part.

[0063] Specifically, in an optional embodiment of the present application, the step S203 of determining part missing information corresponding to each region according to the chassis video stream of each region, pre-stored standard part information, and a pre-constructed model includes:

[0064] S203a: determining a preliminary part detection result of the region according to the chassis video stream of each region and the pre-constructed model.

[0065] In an optional embodiment of the present application, the pre-constructed model can be a deep learning model. In this embodiment, the chassis video stream of each region obtained is input into the deep learning model for result prediction, and the prediction result of the missing parts in the region output by the deep learning model is the preliminary part detection result. The preliminary part detection result can include the installation position of all parts in the region and / or the type of the parts. Then, the installation position or type of the parts is matched and compared with the predicted standard part information, so that the missing part position and the missing part type of the region are obtained as the missing information.

[0066] S203b: According to the preliminary part detection result of the region and the pre-stored standard part information, the Euclidean distance result value between each part in the region and the corresponding pre-stored standard part is determined. When it is detected that the Euclidean distance result value is greater than a preset error threshold, it is determined that the pre-stored standard part is a missing part, and the missing position and the missing type are determined according to the pre-stored standard part information.

[0067] S203c: The missing position of all missing parts is determined as the missing part position of the region, and the missing type of all missing parts is determined as the missing part type of the region.

[0068] In this embodiment, the part detection position in the preliminary part detection result is subjected to Euclidean distance judgment with the position of the pre-stored standard part. If the calculated Euclidean distance result value is greater than a preset error threshold, for example, the preset error threshold is 2 mm, it indicates that the position of the pre-stored standard part has a missing part at the corresponding position on the bottom, and the missing part type is the type of the pre-stored standard part. In this embodiment, the Euclidean distance refers to the relative Euclidean distance of the coordinates of the center of the part with respect to the coordinates of a fixed point of the region where the center of the part is located, for example, the coordinates of the top-left corner.

[0069] Based on the above embodiments, in an optional embodiment of the present application, when the position information of the missing part can be directly detected by using the deep learning model, but the category of the missing part is unknown. In step S203, the part missing information corresponding to each region is determined according to the chassis video stream of each region, the pre-stored standard part information and the pre-constructed model, including:

[0070] S203d: Determine the preliminary part detection result of the region according to the chassis video stream of each region and the pre-constructed model.

[0071] S203e: Determine the missing part position of the region according to the preliminary part detection result of the region. S203f: Determine the Euclidean distance result value between each missing part and each pre-stored standard part according to the pre-stored standard part information and the missing part position of the region, and determine the pre-stored standard part corresponding to the smallest Euclidean distance result value as the missing part type of the missing part.

[0072] In this embodiment, the pre-constructed model can also be a deep learning model, and the preliminary part detection result is different from the above-embodied example in that the preliminary detection result only includes the position information of the missing part. Therefore, the content related to the deep learning model will not be described here again.

[0073] S203g: According to the pre-stored standard part information and the missing part type of all missing parts, the missing part position and the missing part type of the region are determined.

[0074] In this embodiment, the preliminary part detection result only includes the position information of the missing part, and does not include the type of the missing part. At this time, the Euclidean distance result value between the position of the missing part and each pre-stored standard part in the pre-stored standard part information is also calculated. The pre-stored standard part with the minimum Euclidean distance result value can be considered as the missing part whose position information is matched, and the category of the pre-stored standard part corresponding to the minimum Euclidean distance result value is recorded as the missing part type.

[0075] S204: According to all part missing information, all chassis video streams, and detection configuration parameters, the chassis splicing and detection result fusion processing are performed to obtain a target detection result.

[0076] In this embodiment, the chassis splicing and detection result fusion processing refers to the process of splicing the pictures in the chassis video stream into a complete chassis picture and adding the detection result to the complete chassis picture, and the complete chassis picture containing the detection result can be used as the target detection result.

[0077] In an optional embodiment of the present application, the detection configuration parameters include a preset picture overlap ratio. Correspondingly, step S204 includes:

[0078] S204a: Height correction processing is performed on the first ROI picture and the second ROI picture of each region to obtain a first to-be-spliced ROI picture and a second to-be-spliced ROI picture with consistent heights.

[0079] In this embodiment, the height correction processing refers to the process of processing the first ROI and the second ROI picture to have the same height size. For example, the height of the first ROI picture of a certain region is 50 mm, and the height size of the second ROI picture is 60 mm. At this time, the height of the first ROI picture needs to be modified to 60 mm.

[0080] Specifically, in an optional embodiment of the present application, step S204a includes: background brightness filling processing is performed on the first ROI picture and the second ROI picture of each region according to the maximum region height to obtain a first to-be-spliced ROI picture and a second to-be-spliced ROI picture with consistent heights.

[0081] In the embodiment, each region includes two sub-regions, corresponding to the first ROI picture and the second ROI picture respectively. According to the sub-region corresponding to the first ROI picture taken by the left camera and the sub-region corresponding to the second ROI picture taken by the right camera, if the heights of the two sub-regions are inconsistent, the height of the region is determined as the height of the larger one, and the region that needs to be supplemented is filled with the background brightness. For example, if the background brightness is black, the region is filled with black background, and if the background is white, the region is filled with white. In this way, even if one camera is blocked, the size of the unified and well-aligned splicing can be achieved.

[0082] S204b: determining the width value and the height value of the reference picture according to the first ROI picture to be spliced and the second ROI picture to be spliced of all regions.

[0083] In the embodiment, the reference picture can be a picture template for the first ROI picture to be spliced and the second ROI picture to be spliced, or a standard size picture to be obtained after splicing.

[0084] Specifically, in an optional embodiment of the present application, step S204b includes:

[0085] Step A: determining a region width value according to the first ROI picture to be spliced and the second ROI picture to be spliced of each region, comparing all region width values, and determining the maximum value in all region width values as the width value of the reference picture.

[0086] Step B: determining a first ROI picture height accumulation value according to the first ROI picture of all regions, and determining a second ROI picture height accumulation value according to the second ROI picture of all regions.

[0087] Step C: when it is detected that the first ROI picture height accumulation value is greater than the second ROI picture height accumulation value, the first ROI picture height accumulation value is determined as the height value of the reference picture.

[0088] On the basis of the above-mentioned embodiments, in the present embodiment, the height values of the first ROI pictures corresponding to each region on the chassis are accumulated in order from the front to the rear of the vehicle, and the accumulated value of the height values of the first ROI pictures is obtained as the picture size of the chassis from the front to the rear of the vehicle photographed by the left camera. Similarly, the accumulated value of the height values of the second ROI pictures is obtained as the picture size of the chassis from the front to the rear of the vehicle photographed by the right camera. In order to splice the pictures photographed by the left camera and the right camera into a complete chassis picture, the height value of the reference picture needs to be selected as the height value of the picture with the larger size. That is, when it is detected that the accumulated value of the height of the first ROI picture is greater than the accumulated value of the height of the second ROI picture, the accumulated value of the height of the first ROI picture is selected as the height value of the reference picture. Similarly, the width value of the reference picture is selected as the width value of the reference picture by selecting the maximum width value from the width values corresponding to each region.

[0089] S204c: The first ROI pictures to be spliced of all regions are spliced according to the height value of the reference picture in a first alignment manner to obtain a first spliced picture, and the second ROI pictures to be spliced of all regions are spliced according to the height value of the reference picture in a second alignment manner to obtain a second spliced picture.

[0090] In the present embodiment, still taking the left and right cameras as an example, it is assumed that the first ROI pictures to be spliced are photographed by the left camera, and the second ROI pictures to be spliced are photographed by the right camera. The first ROI pictures to be spliced photographed by the left camera are spliced in a right-aligned manner to obtain a first spliced picture, which is a picture of half of the chassis from the front to the rear of the vehicle. The second ROI pictures to be spliced photographed by the right camera are spliced in a left-aligned manner to obtain a second spliced picture, which is another picture of half of the chassis from the front to the rear of the vehicle.

[0091] S204d: The first spliced picture and the second spliced picture are spliced according to the width value of the reference picture in a preset picture overlap ratio in the width direction of the chassis to obtain a target spliced picture including multiple regions of the chassis.

[0092] Figure 3 A schematic diagram of the target spliced picture provided by the embodiments of the present application.

[0093] As Figure 3As shown, in this embodiment, the video stream acquisition device 101 is taken as an example of two cameras, i.e., a left camera and a right camera. Since the sum of the width direction shooting range sizes of the cameras is greater than the width size of the chassis, the picture taken by the left camera and the picture taken by the right camera have an overlapping part near the middle line of the chassis from the front to the rear direction. The preset picture overlap ratio can be understood as the proportion of the overlapping part in the picture. For example, the width of the chassis is two meters, and the shooting range of the left camera and the right camera is 1.1 meters. At this time, it can be known that the width of the overlapping part is 0.1 meters, and the corresponding preset picture overlap ratio is 0.1:1.1.

[0094] In this embodiment, when the first to-be-stitched picture and the second to-be-stitched picture are stitched into the target stitched picture, the overlapping area of the stitching part of the first to-be-stitched picture and the second to-be-stitched picture can be adjusted according to the preset image overlap ratio, so that the width size of the target stitched picture is consistent with the width size of the chassis.

[0095] S204e: The target stitched picture and all the missing part information are subjected to detection result information fusion processing to obtain a target detection result.

[0096] In this embodiment, the missing part information can be text information or color annotation information. The missing part information fusion processing with the target stitched picture means that the missing part information is added to the target stitched picture, so that the missing part detection result can be more intuitively observed by the human. In an optional embodiment of the present application, the missing part detection information can also include the distribution of each missing part in each region.

[0097] In summary, the chassis missing part detection method provided in the embodiments of the present application acquires the initial chassis data, determines the detection configuration parameters according to the initial chassis data, controls the light source 102 to light the chassis according to the detection configuration parameters, controls at least two video stream acquisition devices 101 to collect video streams of each region on the moving chassis at the same time, obtains chassis video streams of multiple regions of the chassis, performs preliminary detection on the chassis according to the chassis video stream of each region, the pre-stored standard part information and the pre-constructed model, determines the missing part information corresponding to each region, and performs chassis stitching and detection result fusion processing according to all the missing part information, all the chassis video streams and the detection configuration parameters to obtain a target detection result. The entire detection process is less affected by human factors, and the detection of the chassis missing part can be automatically completed during the chassis movement to obtain the detection result, thereby improving the detection efficiency and the stability of the detection result.

[0098] Meanwhile, the fusion of the picture and the detection result information is taken as the target detection result, and the distribution of each missing part in each region of the chassis is recorded and summarized, so that the detection result can be more intuitively displayed to the human, and the human can be facilitated to observe.

[0099] Meanwhile, the missing parts are compared with the pre-stored standard parts by using the Euclidean distance to obtain accurate position information and types of the missing parts, so that the detection result is more accurate.

[0100] Meanwhile, different exposure times are adjusted for different regions on the chassis divided according to different card plate materials, so that each frame of picture in the video stream captured by the video stream acquisition device 101 is clearer, better detection materials are provided for subsequent missing detection, and the accuracy of the detection result is improved.

[0101] Based on the above embodiments, the chassis missing detection method provided in an optional embodiment of the present application further includes, after step S204:

[0102] S205: Display processing is performed on the target detection result.

[0103] In this embodiment, the display processing of the target detection result can be performed by sending the target detection result to a corresponding terminal device, which can be a back display of the production line, a mobile phone, a computer or other device with display function of the back office personnel. The specific display method can be selected according to the actual situation.

[0104] In summary, the chassis missing detection method provided in the embodiments of the present application further performs display processing on the target detection result, so that the chassis missing detection result can be viewed more timely and conveniently by manual.

[0105] Figure 4 The structure diagram of the chassis missing detection device provided in the embodiments of the present application is shown.

[0106] The device is applied to the controller 103 in the chassis missing detection system as shown in Figure 1 The chassis missing detection system includes at least two video stream acquisition devices 101, a plurality of light sources 102 and the controller 103. The connection relationship and purpose of each component in the chassis missing detection system have been described in the embodiments as shown in Figure 1 Therefore, the embodiments will not be described here.

[0107] As shown in Figure 4 The chassis missing detection device includes an acquisition module 41, a detection module 42 and a splicing and fusion module 43.

[0108] The acquisition module 41 is configured to acquire initial chassis data and determine detection configuration parameters according to the initial chassis data.

[0109] The acquisition module 41 is further configured to control the at least two video stream acquisition devices 101 and the plurality of light sources 102 to start according to the detection configuration parameters, so as to obtain chassis video streams of a plurality of regions of the chassis.

[0110] The detection module 42 is configured to determine part missing information corresponding to each region according to the chassis video stream of each region, pre-stored standard part information, and a pre-constructed model.

[0111] The splicing and fusion module 43 is configured to perform chassis splicing and fusion processing according to all part missing information, all chassis video streams, and detection configuration parameters, to obtain a target detection result.

[0112] In an optional embodiment of the present application, the detection configuration parameters include a shooting exposure time corresponding to each region of the chassis; accordingly, the acquisition module 41 is specifically configured to: when it is detected that one region of the chassis enters the shooting range of at least two video stream acquisition devices 101, control the at least two video stream acquisition devices 101 and the plurality of light sources 102 to collect a first initial chassis video stream and a second initial chassis video stream of the region according to the shooting exposure time corresponding to the region; determine a first region of interest (ROI) picture corresponding to the region according to the first initial chassis video stream of the region, and determine a second ROI picture corresponding to the region according to the second initial chassis video stream of the region; and determine the first ROI picture and the second ROI picture as the chassis video stream of the region.

[0113] In an optional embodiment of the present application, the detection module 42 is further specifically configured to: determine a preliminary part detection result of the region according to the chassis video stream of each region and the pre-constructed model; determine a Euclidean distance result value between each part in the region and a corresponding pre-stored standard part according to the preliminary part detection result of the region and the pre-stored standard part information, and when it is detected that the Euclidean distance result value is greater than a preset error threshold, determine that the pre-stored standard part is a missing part, and determine a missing position and a missing type according to the pre-stored standard part information; determine the missing position of all missing parts as the missing part position of the region, and determine the missing type of all missing parts as the missing part type of the region.

[0114] In an optional embodiment of the present application, the detection module 42 is further specifically configured to: determine a preliminary part detection result of the region according to the chassis video stream of each region and the pre-constructed model; determine a missing part position of the region according to the preliminary part detection result of the region; determine a Euclidean distance result value between each missing part and each pre-stored standard part according to the pre-stored standard part information and the missing part position of the region, and determine a pre-stored standard part corresponding to the smallest Euclidean distance result value as a missing part type of the missing part; and determine the missing part position and the missing part type of the region according to the pre-stored standard part information and the missing part type of all missing parts.

[0115] In an optional embodiment of the present application, the detection configuration parameter comprises a preset picture overlap ratio; accordingly, the splicing and fusion module 43 is specifically configured to: perform height correction processing on the first ROI picture and the second ROI picture of each region to obtain a first to-be-spliced ROI picture and a second to-be-spliced ROI picture with consistent heights; determine a width value and a height value of a reference picture according to the first to-be-spliced ROI picture and the second to-be-spliced ROI picture of all regions; perform splicing processing on the first to-be-spliced ROI picture of all regions according to the height value of the reference picture in a first alignment manner to obtain a first spliced picture, and perform splicing processing on the second to-be-spliced ROI picture of all regions according to the height value of the reference picture in a second alignment manner to obtain a second spliced picture; perform splicing processing on the first spliced picture and the second spliced picture according to the width value of the reference picture in a preset picture overlap ratio in the chassis width direction to obtain a target spliced picture comprising multiple regions of the chassis; and perform detection result information fusion processing on the target spliced picture and all part missing information to obtain a target detection result.

[0116] In an optional embodiment of the present application, the splicing and fusion module 43 is specifically configured to: perform background brightness filling processing on the first ROI picture and the second ROI picture of each region according to a maximum region height value to obtain a first to-be-spliced ROI picture and a second to-be-spliced ROI picture with consistent heights.

[0117] In an optional embodiment of the present application, the splicing and fusion module 43 is specifically configured to: determine a region width value according to the first to-be-spliced ROI picture and the second to-be-spliced ROI picture of each region, perform comparison processing on all region width values, and determine a maximum value in the all region width values as a width value of a reference picture; determine a first ROI picture height accumulation value according to all first ROI pictures, and determine a second ROI picture height accumulation value according to all second ROI pictures; and when it is detected that the first ROI picture height accumulation value is greater than the second ROI picture height accumulation value, determine the first ROI picture height accumulation value as a height value of the reference picture.

[0118] The chassis missing detection device provided in the present embodiment can be used to execute the technical solutions of the above method embodiments, and has similar implementation principles and technical effects, which will not be described here again.

[0119] Figure 5 A hardware structure schematic diagram of the chassis missing detection device provided in the present embodiment is shown in FIG. 1, which comprises at least one processor 501 and a memory 502. Figure 5

[0120] The memory 502 is configured to store computer execution instructions.

[0121] ​The processor 501 is configured to execute the computer-executable instructions stored in the memory 502, so as to implement the steps involved in the above method embodiments. Details can be referred to the related description in the foregoing method embodiments.

[0122] Optionally, the memory 502 can be independent or integrated with the processor 501.

[0123] When the memory 502 is independently arranged, the device further includes a bus 503 configured to connect the memory 502 and the processor 501.

[0124] The embodiment of the present application further provides a computer readable storage medium, and the computer readable storage medium stores computer-executable instructions. When the processor executes the computer-executable instructions, the tray missing detection method is implemented.

[0125] The embodiment of the present application further provides a computer program product, and the computer program product includes a computer program. When the processor executes the computer program, the tray missing detection method is implemented.

[0126] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented in other manners. For example, the apparatus embodiments described above are merely schematic; for example, the division of the above modules is merely a logical function division; actual implementation can be in another manner, for example, multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection can be indirect coupling or communication connection through some interfaces, devices or modules, and can be electrical, mechanical or in other forms.

[0127] The modules illustrated as separate components can or can not be physically separate, and the components illustrated as modules can or can not be physical units, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to implement the embodiment of the present application.

[0128] In addition, each functional module in each embodiment of the present application can be integrated in one processing unit, or each module can be physically present alone, or two or more modules can be integrated in one unit. The unit composed of the above modules can be realized in the form of hardware, or in the form of hardware plus software functional unit.

[0129] The integrated modules realized in the form of software function modules can be stored in a computer readable storage medium. The software function modules are stored in a storage medium, and include a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute part of steps of the method of various embodiments of the present application.

[0130] It should be understood that the processor described above can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the application can be directly embodied as hardware processor execution, or executed by a combination of hardware and software modules in the processor.

[0131] The memory can include a high-speed RAM memory, and can also include a non-volatile storage NVM, for example at least one disk memory, and can also be a U disk, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk, etc.

[0132] The bus can be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, the bus in the drawings of the present application does not limit to only one bus or one type of bus.

[0133] The storage medium described above can be realized by any type of volatile or non-volatile storage device or their combination, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0134] An exemplary storage medium is coupled to the processor such that the processor can read information from, and can write information to, the storage medium. Of course, the storage medium can be a part of the processor. Consistent with the teachings provided herein, the processor and the storage medium can be implemented as a system-on-a-chip (SOC) where the processor and the storage medium are integrated into a single chip or as part of a system on package (SOP) where the processor and the storage medium are integrated on the same package. Alternatively, the processor and the storage medium can be implemented separately.

[0135] It is to be understood that all or part of the steps of various embodiments of the methods described above can be carried out by program instructions. Such program instructions can be stored in a machine readable medium, which can include any mechanism for storing information, including magnetic storage medium, optical storage medium, and the like. Such equipment can include a processor to execute the program instructions.

[0136] The technical solutions described above are used to illustrate the present application, but not to limit it; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some or all of the technical features can be replaced by equivalent features; and these modifications or replacements do not cause the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A carton underfill detection system, characterized by, include: At least two video stream acquisition devices, multiple light sources, and controllers; At least two video stream acquisition devices are arranged side by side with intervals, and the shooting direction of each video stream acquisition device is facing the chassis; Multiple light sources are arranged in a rectangular pattern around the at least two video stream acquisition devices; The controller is communicatively connected to the at least two video stream acquisition devices; The controller is used to determine the missing parts information for each region based on the chassis video streams collected by the at least two video stream acquisition devices for multiple regions, and according to the chassis video streams for each region, pre-stored standard parts information and pre-constructed models. Based on all missing parts information, all chassis video streams, and inspection configuration parameters, chassis splicing and inspection result fusion processing are performed to obtain target inspection results covering the entire chassis.

2. The underfill detection system of claim 1, wherein The number of video stream acquisition devices is two. The two video stream acquisition devices are cross-calibrated and aligned in the horizontal and vertical directions. The horizontal field of view of the two video stream acquisition devices covers the width of the chassis. The light source is a bar light, and there are 10 bars of light, with the angle of each bar light adjustable from 0 to 360 degrees.

3. A method of underfill detection, the method comprising: A controller applied to the chassis missing parts detection system as described in claim 1, wherein the chassis missing parts detection system comprises: at least two video stream acquisition devices, multiple light sources, and a controller; The chassis missing parts detection method includes: Acquire initial chassis data, and determine the detection configuration parameters based on the initial chassis data; Control the at least two video stream acquisition devices and multiple light sources to start according to the detection configuration parameters to obtain chassis video streams for multiple areas of the chassis; Based on the chassis video stream, pre-stored standard parts information, and pre-built models for each region, determine the missing parts information for each region; Based on all missing parts information, all chassis video streams, and the aforementioned detection configuration parameters, chassis splicing and detection result fusion processing are performed to obtain the target detection result.

4. The method of claim 3, wherein, The detection configuration parameters include the shooting exposure time corresponding to each area of ​​the chassis; Accordingly, controlling the at least two video stream acquisition devices and multiple light sources to start according to the detection configuration parameters to obtain chassis video streams for multiple areas of the chassis includes: When a region of the chassis is detected to be within the shooting range of the at least two video stream acquisition devices, the at least two video stream acquisition devices and multiple light sources are controlled to acquire the first initial chassis video stream and the second initial chassis video stream of the region according to the shooting exposure time corresponding to the region. Based on the first initial chassis video stream of the region, a first region of interest (ROI) image corresponding to the region is determined, and based on the second initial chassis video stream of the region, a second region of interest (ROI) image corresponding to the region is determined. The first region of interest (ROI) image and the second region of interest (ROI) image are determined as the chassis video stream of the region.

5. The method of claim 3, wherein, The process of determining missing parts information for each region based on the chassis video stream, pre-stored standard parts information, and pre-built models includes: According to the chassis video stream of each region and the pre-constructed model, a preliminary part detection result of the region is determined; According to the preliminary part detection result of the region and the pre-stored standard part information, a Euclidean distance result value between each part in the region and the corresponding pre-stored standard part is determined, and when it is detected that the Euclidean distance result value is greater than a preset error threshold, it is determined that the pre-stored standard part is a missing part, and according to the pre-stored standard part information, a missing position and a missing type are determined. The missing position of all missing parts is determined as the missing part position of the region, and the missing type of all missing parts is determined as the missing part type of the region.

6. The method of claim 3, wherein, According to the chassis video stream of each region and the pre-constructed model, a preliminary part detection result of the region is determined; According to the preliminary part detection result of the region, a missing part position of the region is determined; According to the pre-stored standard part information and the missing part position of the region, a Euclidean distance result value between each missing part and each pre-stored standard part is determined, and the pre-stored standard part corresponding to the smallest Euclidean distance result value is determined as the missing part type of the missing part; According to the pre-stored standard part information and the missing part type of all missing parts, the missing part position and the missing part type of the region are determined. The detection configuration parameter includes a preset picture overlap ratio; 7. The method of claim 3, wherein, Accordingly, the chassis splicing and detection result fusion processing according to all part missing information, all chassis video streams and the detection configuration parameter to obtain a target detection result, comprising: The height of the first ROI picture and the second ROI picture of each region is corrected to obtain a first to-be-spliced ROI picture and a second to-be-spliced ROI picture with consistent heights; The width value and the height value of the reference picture are determined according to the first to-be-spliced ROI picture and the second to-be-spliced ROI picture of all regions; The first to-be-spliced ROI picture of all regions is spliced according to the height value of the reference picture in a first alignment manner to obtain a first spliced picture, and the second to-be-spliced ROI picture of all regions is spliced according to the height value of the reference picture in a second alignment manner to obtain a second spliced picture; The first spliced picture and the second spliced picture are spliced according to the width value of the reference picture in a preset picture overlap ratio in the chassis width direction to obtain a target spliced picture including multiple regions of the chassis; The target spliced picture and all part missing information are subjected to detection result information fusion processing to obtain a target detection result. The height of the first ROI picture and the second ROI picture of each region is corrected to obtain a first to-be-spliced ROI picture and a second to-be-spliced ROI picture with consistent heights, comprising:

8. The method of claim 7, wherein, ​ The first ROI picture and the second ROI picture of each region are filled with background brightness according to the region height maximum, to obtain a first to-be-stitched ROI picture and a second to-be-stitched ROI picture with consistent heights.

9. The method of claim 7, wherein, The width value and the height value of the reference picture are determined according to the first to-be-stitched ROI picture and the second to-be-stitched ROI picture of all regions, and the method comprises the following steps: A region width value is determined according to the first to-be-stitched ROI picture and the second to-be-stitched ROI picture of each region, and all region width values are compared to determine the maximum value of all region width values as the width value of the reference picture; A first ROI picture height accumulation value is determined according to the first ROI picture of all regions, and a second ROI picture height accumulation value is determined according to the second ROI picture of all regions; When it is detected that the first ROI picture height accumulation value is greater than the second ROI picture height accumulation value, the first ROI picture height accumulation value is determined as the height value of the reference picture.

10. A carton underfill detection apparatus, characterized by, The controller is applied to the chassis missing detection system of claim 1, wherein the chassis missing detection system further comprises at least two video stream acquisition devices, a plurality of light sources and the controller. The chassis missing detection device comprises: An acquisition module is configured to acquire chassis initial data and determine detection configuration parameters according to the chassis initial data. The acquisition module is further configured to control the at least two video stream acquisition devices and the plurality of light sources to start according to the detection configuration parameters, to obtain chassis video streams of a plurality of regions of the chassis. A detection module is configured to determine part missing information corresponding to each region according to the chassis video stream of each region, pre-stored standard part information and a pre-constructed model. A stitching and fusion module is configured to perform chassis stitching and fusion processing according to all part missing information, all chassis video streams and the detection configuration parameters, to obtain a target detection result.

Images