Vehicle window positioning detection method and vehicle interior inspection system
By detecting the window opening and closing degree and calculating the three-dimensional world coordinates, the problem of difficulty in positioning the window in the automatic vehicle inspection system is solved, and the accuracy and efficiency of in-vehicle inspection are improved.
Patent Information
- Application Number
- CN202410897784.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-05
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2044-07-05
AI Technical Summary
In the prior art, it is difficult for the automatic inspection system of a vehicle in a security inspection site to accurately obtain the physical coordinates of the window opening position, which affects the accurate movement of the robot arm and the efficiency of in-vehicle inspection.
The depth camera collects color images and depth images on the side of the vehicle, detects the window opening and closing degree, and calculates the three-dimensional world coordinates of the window opening area through the histogram statistics and target detection model of the depth information, providing the target position of the robotic arm movement .
Automatic positioning and physical coordinate calculation of window opening areas are realized, the accuracy and efficiency of automatic inspection in the car are improved, and manual intervention is reduced.
Smart Images

Figure CN118447096B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of digital image processing, and more specifically, to a vehicle window positioning detection method and an in-vehicle inspection system. Background Art
[0002] At the security inspection site, it is necessary to conduct a safety inspection on the vehicles passing through the passage, which may include checking the number of passengers including the driver and checking whether there are any dangerous items in the cabin. If it is a manual inspection, the security personnel need to wait for a long time on the lane for the passing vehicles, and after opening the car door, the inspectors need to count the number of people and check whether there are any dangerous items in the cabin, and when taking on-site photos of the situation inside the car, the inspectors need to take photos with handheld cameras. If an automatic inspection system is used to inspect the interior of the vehicle staying in the lane, the automatic inspection system includes a multi-axis movable mechanical arm arranged on the lane and a monitoring camera arranged on the mechanical arm. The automatic inspection system can drive the monitoring camera to extend from the open window of the vehicle into the vehicle through the moving mechanical arm, and then the situation inside the vehicle can be viewed and photographed through the monitoring camera. The automatic inspection method is more convenient and intelligent than the aforementioned manual inspection method. However, if the aforementioned automatic inspection method is to be implemented, it is necessary to know the actual physical coordinates of the window position of the vehicle to provide a target position for the subsequent movement of the mechanical arm. Summary of the invention
[0003] In view of the existing technology, the technical problem solved by the present application is to provide a vehicle window positioning detection method and an in-vehicle inspection system that can automatically locate the real physical coordinates of the window opening area of the vehicle window through image vision methods to provide the target position for subsequent automatic inspection.
[0004] In order to solve the above technical problems, in a first aspect, the present application provides a vehicle window positioning detection method, which comprises:
[0005] Obtain image data F of the side of the vehicle collected by the depth camera, wherein the image data F includes a color image F 1 and the depth image F 2 ;
[0006] Based on the color image F 1 Detecting the opening degree Deg of the vehicle window when it is open or detecting the opening degree Deg of the vehicle window when it is open based on the image data F;
[0007] When the opening degree is greater than the first threshold, the depth image F 2 Performing histogram statistics of depth information to obtain the distance D from the vehicle body to the depth camera; and,
[0008] The three-dimensional world coordinate value from the center point of the window area to the depth camera is calculated based on the distance D from the vehicle body to the depth camera.
[0009] In a possible implementation, based on the color image F 1 Detect the opening and closing degree Deg of the car window when it is opened, including:
[0010] Based on the first target detection model, the color image F is detected 1 The closest color image F among all the windows in 1 The center of the middle window target Mid box ;
[0011] Based on the second target detection model, the color image F is detected 1 The glass targets of all car windows are obtained to obtain the glass target set G glass ;
[0012] In the glass target set G glass Find the target Mid that meets the first preset condition box Glass Target Mid glass ;
[0013] If, in the glass target set G glass Failed to find a glass target frame that meets the first preset condition Mid glass ;Then the window opening degree Deg=100%;
[0014] Otherwise, according to the middle window target Mid box and glass target Mid glass Calculate the window opening degree Deg: window opening degree Deg = 1-(H Midglass / H Midbox );
[0015] Among them, H Midbox For the window target Mid box The height of the target box, H Midglass For glass target Mid glass The height of the target box.
[0016] In a possible implementation, detecting the opening degree Deg of the vehicle window when it is opened based on the image data F includes:
[0017] Based on the first target detection model, the color image F is detected 1 The closest color image F among all the windows in 1 The center of the middle window target Mid box ;
[0018] For the depth image F 2 Perform histogram statistics of depth information to obtain the distance D from the vehicle body to the depth camera;
[0019] The distance D is used as the second threshold for the depth image F 2 Perform binarization processing to obtain a binary image;
[0020] Creating a first structure element and a second structure element for an image erosion operation, wherein the size of the first structure element is larger than the second structure element;
[0021] Sequentially using the first structuring element and the second structuring element to perform an erosion operation on the binary image to obtain a first image;
[0022] Performing a connected domain solving operation on the first image to obtain a second image including a plurality of connected domains;
[0023] Calculate the minimum bounding rectangle of all connected domains, and select the minimum bounding rectangle whose width meets the second preset condition from all the minimum bounding rectangles to form a minimum bounding rectangle set G Rect ;
[0024] In the minimum bounding rectangle set G Rect Select the closest depth image F 2 The minimum enclosing rectangle of the central area Mid Rect ;
[0025] According to the middle window target Mid box and the minimum enclosing rectangle Mid Rect Calculate the window opening degree Deg: Window opening degree Deg = H grect / H Midbox ;
[0026] Among them, H Midbox For the window target Mid box Height, H grect is the minimum enclosing rectangle Mid Rect height.
[0027] In a possible implementation, the color image F is detected based on the first target detection model. 1 The closest color image F among all the windows in 1 The center of the middle window target Mid box ,include:
[0028] Based on the first target detection model, the color image F is detected 1 All window target boxes in ;
[0029] Calculate the area of all window targets, sort all area values from large to small, and select the first n window target frames after sorting to form the window target set G box ; Calculate the window target set G box The center point of each window target box in the color image F1 The Euclidean distance of the center point;
[0030] Select the window target corresponding to the smallest Euclidean distance value as the middle window target Mid box .
[0031] In one possible implementation, the middle window target Mid is set box The horizontal coordinates of the upper left corner and the lower right corner of the target box are and , assuming that the glass target set G glass A glass target glass i The horizontal coordinates of the upper left corner and the lower right corner of the target box are and ;
[0032] Mid window target box and glass target glass i When the following first preset condition is met, the glass target set G glass A glass target glass i To find the glass target Mid glass ;
[0033] Among them, the first preset condition is:
[0034] and The value of is greater than a third threshold;
[0035] in, To obtain the maximum value, To perform the minimum value operation.
[0036] In one possible implementation, any minimum bounding rectangle R ecti Belongs to the set G Rect The second precondition that needs to be met is:
[0037] W Recti >T W And W Recti <0.8·W F2
[0038] Among them, W Recti is the minimum external target box R ecti Width, W F2 is the depth image F 2 Width, T W A fourth threshold value regarding width is set.
[0039] In a possible implementation, the depth image F 2Perform histogram statistics of depth information to obtain the distance D from the vehicle to the depth camera, including:
[0040] The threshold range of the depth information when the histogram is counted is set according to the position of the depth camera installed on the lane and the width of the lane [H 1 ,H 2 ], where H 1 <H 2 ;
[0041] For the depth image F 2 belongs to the threshold range [H 1 ,H 2 ] to perform histogram statistics on the number of pixels corresponding to each depth information value;
[0042] The depth information value corresponding to the maximum value of the vertical axis of the histogram is taken as the distance D from the vehicle body to the depth camera.
[0043] In a possible implementation, when the opening degree = 100%, the three-dimensional world coordinate value from the center point of the window area to the depth camera is calculated based on the distance D from the vehicle body to the depth camera, which is:
[0044] ; ;
[0045] in, , , and The intrinsic parameter matrix K from the depth camera; , are the horizontal and vertical coordinates of the origin of the image physical coordinate system in the image pixel coordinate system respectively; , is the horizontal and vertical coordinates of the center point of the window area in the image, that is, the obtained middle window target Mid box The horizontal and vertical coordinates of the center point in the image; and are the focal length of the depth camera in the horizontal and vertical directions respectively; It is the 3D world coordinate value from the center point of the windowing area to the depth camera.
[0046] In a possible implementation, when the opening degree is ≠100%, the three-dimensional world coordinate value from the center point of the window area to the depth camera is calculated based on the distance D from the vehicle body to the depth camera, which is:
[0047] ; ;
[0048] in, , , and The intrinsic parameter matrix K from the depth camera; , are the horizontal and vertical coordinates of the origin of the image physical coordinate system in the image pixel coordinate system respectively; , is the horizontal and vertical coordinates of the center point of the window opening area in the image. The center point of the window opening area is the minimum circumscribed rectangle R ecti The horizontal and vertical coordinates of the center point in the image; and are the focal length of the depth camera in the horizontal and vertical directions respectively; It is the 3D world coordinate value from the center point of the windowing area to the depth camera.
[0049] In a second aspect, the present application provides an in-vehicle inspection system, which includes:
[0050] The depth camera set on the safety island on one side of the lane is used to collect the side image of the vehicle parked in the preset position of the lane to obtain image data F, wherein the image data F includes a color image F 1 and the depth image F 2 ;
[0051] A mechanical arm disposed on the same side of the lane as the depth camera can move forward and backward relative to the vehicle on the lane and can move up and down relative to the ground;
[0052] An inspection device disposed on the mechanical arm;
[0053] A first processing module, which is communicatively connected to the depth camera and is used to execute the steps of the vehicle window positioning detection method to obtain a three-dimensional world coordinate value from the center point of the window opening area to the depth camera; and
[0054] The second processing module is communicatively connected to the first processing module and is used to convert the three-dimensional world coordinates from the center point of the window area to the depth camera obtained by the first processing module into coordinate values in the robotic arm coordinate system.
[0055] The beneficial effects of the vehicle window positioning detection method and detection device provided in the present application are: based on the depth camera, the vehicle window opening degree is automatically detected through image vision methods, and the center of the window opening area that meets the opening and closing threshold is calculated in the three-dimensional world coordinate system under the depth camera. The physical coordinate position of the center of the window opening area can be automatically located based on the image vision method to provide a target position for the subsequent movement of the robotic arm, so as to assist in the subsequent completion of the automatic inspection function in the vehicle. BRIEF DESCRIPTION OF THE DRAWINGS
[0056] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0057] Figure 1 A flowchart of the steps of the vehicle window positioning detection method according to an embodiment of the present application;
[0058] Figure 2 It is a color image of image data of the side of the vehicle captured by the depth camera of an embodiment of the present application.
[0059] Figure 3 This is a flow chart of the method steps for detecting the opening degree Deg of a vehicle window when it is opened based on a color image F1 according to an embodiment of the present application.
[0060] Figure 4 For the embodiments of this application Figure 2 Detected middle window target Mid box Schematic diagram of the results.
[0061] Figure 5 For the embodiments of this application Figure 2 The glass target set G of color images glass Schematic diagram of the results.
[0062] Figure 6 This is a flow chart of the method steps for detecting the opening degree Deg of a vehicle window when it is opened based on image data F according to an embodiment of the present application.
[0063] Figure 7 A schematic diagram of a scene of a vehicle in a lane and a depth camera on the lane according to an embodiment of the present application.
[0064] Figure 8 This is a histogram statistical result of the depth information of a depth image when the depth information is within the threshold range [0.5, 2.5] and the statistical interval of the histogram is 5 cm in an embodiment of the present application.
[0065] Fig. 9 The depth image F of the embodiment of the present application 2 Schematic diagram of the result of the binarization operation.
[0066] Fig.10 For the embodiment of this application Fig. 9 The resulting image after the corrosion operation.
[0067] Fig.11 A schematic diagram of a depth camera coordinate system and a robotic arm coordinate system on a lane according to an embodiment of the present application. DETAILED DESCRIPTION
[0068] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0069] It should be noted that when an element is referred to as being "fixed to" or "disposed on" another element, it can be directly on the other element or indirectly on the other element. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or indirectly connected to the other element.
[0070] It should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.
[0071] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.
[0072] The vehicle window positioning detection method and the in-vehicle inspection system of the present application are now described in detail with reference to the accompanying drawings.
[0073] Please refer to Figure 1 The vehicle window positioning detection method provided in the embodiment of the present application is used to locate the position of the window opening area of the vehicle, which includes the following steps S100 to S400:
[0074] Step S100: Acquire image data F of the side of the vehicle collected by the depth camera, wherein the image data F includes a color image F 1 and the depth image F 2 ;
[0075] Step S200: Based on the color image F 1 Detecting the opening degree Deg of the vehicle window when it is open or detecting the opening degree Deg of the vehicle window when it is open based on the image data F;
[0076] Step S300: When the opening degree is greater than a first threshold, the depth image F 2 Performing histogram statistics of depth information to obtain the distance D from the vehicle body to the depth camera; and
[0077] Step S400: Calculate the three-dimensional world coordinate value from the center point of the window area to the depth camera based on the distance D from the vehicle body to the depth camera.
[0078] It is worth noting that the vehicle window positioning detection method is used to position the window area of a vehicle parked in a lane. The front of the vehicle is the front, the rear of the vehicle is the rear, and the side with the window is the side. That is, the side of the vehicle in step S100 refers to the side with the window. Figure 2 A color image F of the image data F of the side of the vehicle collected by the depth camera is shown 1 When the depth camera collects images, it can obtain color image data and depth image data at the same time. 1 and the depth image F 2 The two image data are aligned, that is, the color image F 1 and the depth image F 2 The same pixel coordinates correspond to the same position in the real scene.
[0079] In the above-mentioned vehicle window positioning detection method, the depth camera automatically detects the window opening degree through image vision methods and calculates the three-dimensional world coordinate system of the center of the window opening area that meets the opening and closing threshold under the depth camera. The physical coordinate position of the center of the window opening area can be automatically located based on the image vision method to provide a target position for the subsequent movement of the robotic arm, so as to assist in the subsequent automatic inspection inside the vehicle.
[0080] In the first application embodiment, please refer to Figure 3 For step S200, based on the color image F 1 Detecting the opening degree Deg of the vehicle window when it is opened specifically includes the following steps S211 to S216.
[0081] Step S211: Detecting a color image F based on the first object detection model 1 The closest color image F among all the windows in 1 The center of the middle window target Mid box , specifically:
[0082] Based on the first target detection model, the color image F is detected 1 All window target boxes in ;
[0083] Calculate the area of all window targets, sort all area values from large to small, and select the first n window target frames after sorting to form the window target set Gbox ;
[0084] Calculate the window target set G box The center point of each window target box in the color image F 1 The Euclidean distance of the center point;
[0085] Select the window target corresponding to the smallest Euclidean distance value as the middle window target Mid box .
[0086] It is worth noting that when the embodiment of the present application processes a private car parked in a lane, the first target detection model may be, but is not limited to, a YOLOX or YOLOX5 deep learning network model. When training the first target detection model, vehicle window data is collected, and the YOLOX network model is trained using any existing training method to obtain a first target detection model for detecting vehicle window targets. 1 Perform target detection to obtain a color image F 1 All the window targets in the target detection. When detecting the target, the target frame information of each window target will be fed back. Assuming that any window detected is Window j , then the vehicle target Window j The target frame information is (C Winxj ,C Winyj ,W Winj ,H Winj ), where C Winxj ,C Winyj ,W Winj ,H Winj They are the window targets Window j The horizontal coordinate of the center point of the target frame, the vertical coordinate of the center point of the target frame, the width of the target frame, and the height of the target frame.
[0087] Understandably, after knowing the horizontal coordinate of the center point of a target frame, the vertical coordinate of the center point of the target frame, the width of the target frame, and the height of the target frame, the coordinates and area of the four corner points of the target frame can be calculated.
[0088] Among them, any window target Window j The area of its target box is W Winj ×H WinjThe vehicle to be inspected is parked at a preset position in the lane, which is planned in advance. The depth camera is also fixed around the preset position. When the vehicle is parked at the preset position, most of the depth camera screen will be occupied by the vehicle target to be inspected. In addition, the driver's window and the rear of the vehicle are generally on both sides of the color image captured by the depth camera. Compared with other windows, the rear passenger window will be relatively concentrated in the middle of the color image. Figure 2 As shown, the color image F 1 The target includes not only the window targets that can be opened, but also the smaller window targets at the rear that cannot be opened. After sorting the areas of all the window targets from large to small, the first n window target frames are selected in order of area value. In this way, the aforementioned smaller window targets that cannot be opened and other smaller interfering targets with similar window shapes can be excluded, so as to obtain the window targets that are really needed.
[0089] It is worth noting that, in general, the driver's cabin area is relatively small compared to the rear cabin area, so it is more appropriate for the surveillance camera to extend from the rear window. In addition, the passengers are mainly concentrated in the rear row, so the embodiment of the present application considers selecting the rear window. 1 The Euclidean distance of the center point is used to measure the distance between the window target and the center of the color image. The window target corresponding to the minimum Euclidean distance value is selected as the middle window target Mid box .in, Figure 4 For Figure 2 Detected middle window target Mid box Schematic diagram of the results.
[0090] Step S212: Detect the color image F based on the second object detection model 1 The glass targets of all car windows are obtained to obtain the glass target set G glass .
[0091] The second target detection model may be, but is not limited to, a YOLOX or YOLOX5 deep learning network model. When training the second target detection model, vehicle window data is collected, and the YOLOX network model is trained using any existing training method to obtain a first target detection model for detecting glass targets on vehicle windows. Similarly, the second target detection network is used to detect the color image F 1 Perform target detection to obtain a color image F 1 The target box information of each glass target will be fed back for each glass target during target detection. Assuming that any glass target detected is Glass j , then the glass target Glass j The target frame information is (C Glassxj,C Glassyj ,W Glassj ,H Glassj ), where C Glassxj ,C Glassyj ,W Glassj ,H Glassj Glass targets j The horizontal coordinate of the center point of the target frame, the vertical coordinate of the center point of the target frame, the width of the target frame, and the height of the target frame.
[0092] Step S213: In the glass target set G glass Find the target Mid that meets the first preset condition box Glass Target Mid glass .
[0093] It can be understood that the window target and the glass target of the window target have a one-to-one correspondence. Set the middle window target Mid box The horizontal coordinates of the upper left corner and the lower right corner of the target box are and , assuming that the glass target set G glass A glass target glass i The horizontal coordinates of the upper left corner and the lower right corner of the target box are and . The target of the middle window is Mid box and glass target glass i When the following first preset condition is met, the glass target set G glass A glass target glass i To find the glass target Mid glass . Among them, the first preset condition is:
[0094] and The value of is greater than the third threshold; wherein, To obtain the maximum value, For the minimum operation, the third threshold may be set to 0.7.
[0095] It is worth noting that in the color image F 1 , an image coordinate system is established with the upper left corner of the image as the origin. The upper left corner of a target frame is the corner of the target frame closest to the origin, and the lower right corner is another corner of the target frame on the same diagonal as the upper left corner. It can be understood that the target frames obtained by target detection are all rectangular frames. The width of the target frame involved in the embodiment of the present application is the length of the target frame in the horizontal axis direction of the image coordinate system, and the height of the target frame involved is the length of the target frame in the vertical axis direction of the image coordinate system.
[0096] Furthermore, if the window and the glass are corresponding, the width of the detected window target frame and the glass target frame will be similar, but the position may be different; in terms of vertical relationship, the height of the two window target frames and the glass target frame will be different due to the different opening and closing degrees of the windows. box Corresponding glass target Mid glass When , we can only rely on the relationship of the horizontal coordinates of the target frame, that is, the horizontal axis coordinate direction.
[0097] Step S214: Determine whether the glass target set G glass Whether the glass target Mid that meets the first preset condition has been found glass If not found, execute step S215, otherwise execute step S216.
[0098] Step S215: Window opening degree Deg=100%.
[0099] Step S216: According to the middle window target Mid box and glass target Mid glass Calculate the window opening degree Deg.
[0100] According to the window target Mid box and glass target Mid glass Calculate the window opening degree Deg, which is window opening degree Deg = 1-(H Midglass / H Midbox ). Among them, H Midbox For the window target Mid box The height of the target box, H Midglass For glass target Mid glass As mentioned above, for whether the window is open or not, the vehicle target and the glass target can be not much different in the horizontal axis direction. In order to better obtain the opening degree, the vertical coordinate relationship between the window target and the glass target is considered.
[0101] It is worth noting that if in the glass target set G glass Failed to find the target Mid that meets the first preset condition box Glass Target Mid glass , indicating the middle window target Mid box The window of the car has been completely lowered and hidden, and the middle window target Mid box In the fully open state, the middle window target Mid box The opening degree of the car window is 100%. If a glass target frame Mid that meets the first preset condition is found glass , then according to the middle window target Midbox and the glass target Mid glass The height of the window is calculated. Figure 5 for Figure 2 The glass target set G of color images glass Schematic diagram of the results.
[0102] In the second application embodiment, please refer to Figure 6 For step S200, the opening degree Deg of the window when the window is opened is detected based on the image data F, which specifically includes the following steps S221 to S229. The second application embodiment and the first application embodiment use different methods to obtain the opening degree.
[0103] Step S221: Detecting a color image F based on the first object detection model 1 The closest color image F among all the windows in 1 The center of the middle window target Mid box .
[0104] Among them, the implementation process of step S221 is the same as the implementation process of step S211 in the above-mentioned first application embodiment.
[0105] Step S222: Depth image F 2 Performing histogram statistics of depth information to obtain the distance D from the vehicle body to the depth camera specifically includes the following steps S2221 to S2223:
[0106] Step S2221: Setting a threshold range of depth information for histogram statistics based on the position of the depth camera installed on the lane and the width of the lane [H 1 ,H 2 ], where H 1 <H 2 ;
[0107] Step S2222: Depth image F 2 belongs to the threshold range [H 1 ,H 2 ] to perform histogram statistics on the number of pixels corresponding to each depth information value;
[0108] Step S2223: The depth information value corresponding to the maximum value of the vertical axis of the histogram is used as the distance D from the vehicle body to the depth camera, that is, the horizontal axis value of the histogram corresponding to the maximum value of the vertical axis of the histogram is used as the distance D from the vehicle body to the depth camera.
[0109] Among them, H 1 ,H 2 Set two thresholds, H 1 =h 1 +t,H 2=0.5·h 2 + H 1 ;h 2 is the width of the lane. Since the depth camera is installed on the safety island on one side of the lane, h 1 is the horizontal distance from the depth camera to the edge of the lane, such as Figure 7 The above t is a set threshold value, which represents the preset parking distance between the vehicle body and the edge of the lane where the depth camera is set when the vehicle is parked at the preset position of the lane. The value of t can be 0.2 meters. 1 and H 2 The unit is meter.
[0110] It can be understood that the depth information of the depth image is subjected to histogram statistics, that is, a histogram is established with the depth information representing the distance information as the horizontal axis and the total number of pixels corresponding to each depth information value as the vertical axis. Figure 8 Indicated when [H 1 ,H 2 ] is in the range of [0.5,2.5] and the statistical interval of the histogram is 5 cm. Since the depth camera collects images of vehicles parked in the channel on the safety island on one side of the lane, and collects the side view of the vehicle, and in the histogram, the threshold range [H 1 ,H 2 ] is mostly the depth information of the vehicle body, and the threshold range [H 1 ,H 2 ] range can exclude some depth information that is not very relevant to the vehicle target. In the histogram statistics, the depth information value corresponding to the maximum value of the vertical axis of the histogram is used as the distance D from the vehicle body to the depth camera. Figure 8 The statistical results of the histogram are as follows: distance D = 1.65 meters.
[0111] Step S223: Using the distance D as the second threshold to analyze the depth image F 2 Perform binarization processing to obtain a binary image.
[0112] Understandably, the depth image F 2 Each pixel in the image represents the distance information from a certain position in the real scene to the depth camera. The acquisition target of the depth camera is the vehicle, and most of the field of view of the depth camera is the vehicle target. Then, using the distance information as the threshold of the binarization operation, the segmentation process can distinguish the window area and the area outside the body.
[0113] Among them, when the distance D is used as the second threshold for binarization operation, the depth image F 2 Pixels whose pixel values are less than the distance D are assigned a value of 0, otherwise the pixel values are assigned a value of 255. Fig. 9 For the depth image F2 Schematic diagram of the result of the binarization operation.
[0114] Step S224: creating a first structure element and a second structure element for image erosion operation, wherein the size of the first structure element is larger than that of the second structure element.
[0115] Step S225: performing an erosion operation on the binary image using the first structuring element and the second structuring element in sequence to obtain a first image.
[0116] The first structural element Can be 15×7, the second structural element It can be 3×3. The larger corrosion structure element is recorded as , used to remove noise from depth image data; smaller erosion structural elements It is used to connect multiple regions that are very close to each other and merge them into a large region. Apply the structural elements one by one , , perform corrosion operation: ,in, Represents an image erosion operation. Fig.10 For Fig. 9 The resulting image after the corrosion operation.
[0117] Step S226: performing a connected domain solving operation on the first image to obtain a second image including a plurality of connected domains. The connected domain solving operation may adopt an existing method.
[0118] Step S227: Calculate the minimum bounding rectangles of all connected domains, and select the minimum bounding rectangle whose width meets the second preset condition from all the minimum bounding rectangles to form a minimum bounding rectangle set G Rect .
[0119] Among them, any minimum enclosing rectangle R ecti Belongs to the set G Rect The second precondition that needs to be met is:
[0120] W Recti >T W And W Recti <0.8·W F2
[0121] Among them, W Recti is the minimum external target box R ecti Width, W F2 is the depth image F 2 Width, T W The fourth threshold value of width is set to 150.openwj <0.8·W F2 , we can exclude the relatively large connected areas representing the area outside the vehicle body in the binarization results, such as Fig.10 The connected area with the largest pixel value at the top being 255. openwj >T W , too small connected areas can be excluded, because as long as the car window is opened, no matter how large the open area is, its width has a certain length.
[0122] Step S228: In the minimum bounding rectangle set G Rect Select the closest depth image F 2 The minimum enclosing rectangle of the central area Mid Rect .
[0123] The minimum bounding rectangle set G Rect The center point of each bounding rectangle in the depth image F 2 The Euclidean distance is calculated from the center point of the image, and the bounding rectangle with the smallest Euclidean distance value is selected as the one closest to the depth image F. 2 The minimum enclosing rectangle of the central area Mid Rect .
[0124] Step S229: According to the middle window target Mid box and the minimum enclosing rectangle Mid Rect Calculate the window opening degree Deg; specifically: window opening degree Deg = H grect / H Midbox Among them, H Midbox For the window target Mid box Height, H grect is the minimum enclosing rectangle Mid Rect height.
[0125] In the above embodiment, for step S300, when the opening degree is greater than the first threshold, the depth image F 2 Perform histogram statistics of depth information to obtain the distance D from the vehicle body to the depth camera. The first threshold value is 0.8. The larger the window opening degree Deg value is, the greater the degree of window opening is. Considering the errors in each detection and calculation process, and considering the large diameter of the mechanical arm, it is usually necessary to ensure that the window opening degree Deg is at least 80% to ensure the safety of the equipment. The implementation process of step S300 is the same as the implementation process of the aforementioned step S222, and will not be repeated here.
[0126] It is worth mentioning that if the opening degree is less than the first threshold, it means that it is not opened enough. At this time, the background monitoring system will control the voice announcer to broadcast a voice similar to "The window opening area is not enough, please open the window further" to remind you to continue opening the window.
[0127] In an embodiment of the application, when the opening degree = 100%, the three-dimensional world coordinate value from the center point of the window area to the depth camera is calculated based on the distance D from the vehicle body to the depth camera, which is:
[0128] ; ;
[0129] Among them, the above formula is based on the three-dimensional vision foundation. It can be understood that it involves the relationship between the image physical coordinate system, the image pixel coordinate system, the camera coordinate system and the three-dimensional world coordinate system. , , and The intrinsic parameter matrix K from the depth camera; the intrinsic parameter matrix K can be obtained by calibrating the depth camera; , are the horizontal and vertical coordinates of the origin of the image physical coordinate system in the image pixel coordinate system respectively; , is the horizontal coordinate and vertical coordinate of the center point of the window opening area in the image data F, that is, the middle window target Mid obtained in step S211 box The horizontal and vertical coordinates of the center point. and They are the focal lengths of the depth camera in the horizontal and vertical directions, respectively. The image physical coordinate system is the CCD image plane of the depth camera, the image pixel coordinate system is a coordinate system established with the upper left corner of the image generated by the depth camera as the origin, the origin of the depth camera coordinate system is the center of the depth camera, the optical axis of the depth camera is the Z axis z_camera of the depth camera, the line perpendicular to the z_camera axis and pointing to the ground is the Y axis y_camera of the depth camera coordinate system, and the line perpendicular to the z_camera axis and the y_camera axis is the X axis of the depth camera coordinate system, as shown in Fig.11 As shown; the optical axis of the depth camera is the central axis of the camera. is the 3D world coordinate value from the center point of the window area to the depth camera, which can also be understood as the middle window target Mid in the real scene box The center position to the coordinate value in the depth camera coordinate system. They are respectively the X-axis coordinate, Y-axis coordinate and Z-axis coordinate in the world coordinate system.
[0130] For step S400, in another embodiment of the application, when the opening degree is ≠100%, the three-dimensional world coordinate value from the center point of the window area to the depth camera is calculated based on the distance D from the vehicle body to the depth camera, which is
[0131] ; ;
[0132] In this embodiment, , is the horizontal coordinate and vertical coordinate of the center point of the window opening area in the image data F. The center point of the window opening area is: the middle window target Mid obtained in step S211 box Remove step S213 to obtain glass target Mid glass The horizontal and vertical coordinates of the center point of the remaining area R in the image data F are box and Mid box Glass area within the targetMid glass If the two are obtained, the remaining area R can be obtained by subtracting them, and the center point of the remaining area R can also be known. Alternatively, the center point of the window area is the minimum bounding rectangle R obtained in step S226. ecti The horizontal coordinate and the vertical coordinate of the center point in the image data F. Other parameters are the same as those in the previous application embodiment.
[0133] Please refer to Fig.11 An embodiment of the present application provides an in-vehicle inspection system that communicates and connects to a monitoring background. The in-vehicle inspection system is used to implement the above-mentioned vehicle positioning detection method steps. The in-vehicle inspection system includes a depth camera arranged on a safety island on one side of a lane, a robotic arm arranged on the same side of the lane as the depth camera, an inspection device arranged on the robotic arm, a first processing module that communicates and connects to the depth camera, and a second processing module that communicates and connects to the first processing module.
[0134] It is worth noting that the lane will plan a preset position for the vehicle to park in advance. For example, a preset frame is marked in the lane as the preset position, and the depth camera and the robotic arm are set on the safety island on the same side of the lane. The depth camera and the robotic arm can be set side by side relative to the front and rear of the vehicle and are set on one side of the preset position at the same time. The depth camera is located on the side of the passing vehicle, that is, on the side of the vehicle with windows; for the designated preset position, when the passing vehicle stops at the preset position, the depth camera is close to the middle of the side of the vehicle. It can be understood that after the depth camera is installed and debugged, it can be ensured that most of the field of view of the image generated by the depth camera is the vehicle target within the preset position.
[0135] The depth camera collects the vehicle target to obtain image data F, wherein the image data F includes a color image F 1 and the depth image F 2. The robot arm can telescopic and retract from the vehicle in the lane and can move up and down relative to the ground. The robot arm is a standard piece, such as a two-axis robot arm, a three-axis robot arm or a multi-axis robot arm in the prior art. The robot arm can telescopic and retract from the vehicle in the lane and can move up and down relative to the ground. The first processing module performs the aforementioned steps of the window positioning detection method to obtain the three-dimensional world coordinate value of the center point of the window opening area to the depth camera, that is, performs all the steps of the aforementioned steps S100 to S400; the second processing module is used to convert the three-dimensional world coordinates of the center point of the window opening area to the depth camera obtained by the first processing module to the coordinate value under the robotic arm coordinate system. The first processing module and the second processing module may be an image processing chip, both of which may be two separate image processing chips or both integrated into one chip. For the second processing module, the implementation process of converting the three-dimensional world coordinates of the center point of the window opening area obtained by the first processing module to the coordinate value under the robotic arm coordinate system is as follows.
[0136] like Fig.11 As shown, the establishment of a mechanical coordinate system: the center of the end of the robot arm is installed on the safety island is the origin of the robot arm coordinate system, the X-axis x_robot_arm of the robot arm coordinate system along the lane's traffic direction (not shown), the Y-axis y_robot_arm of the robot arm coordinate system is pointed at the vehicle direction in the lane with the origin point as the Y-axis y_robot_arm of the robot arm coordinate system, and the straight line of the vertical ground is the Z-axis z_robot_arm of the robot arm coordinate system. Establishment of depth camera coordinate system: the center point of the depth camera is the origin of the depth camera coordinate system, the optical axis of the depth camera is the Z-axis z_camera of the depth camera, the line perpendicular to the z_camera axis and pointing to the ground is the Y-axis y_camera of the depth camera coordinate system, and the lines perpendicular to the z_camera axis and y_camera axis are the X-axis of the depth camera coordinate system; the optical axis of the depth camera is the camera's central axis.
[0137] As mentioned above, after the default vehicle is parked at the preset position, the robot arm is basically facing the center position of the aforementioned window opening area. In this way, the robot arm does not need to move along the lane traffic direction, so the robot arm does not cause displacement movement in x_robot_arm under the established robot arm coordinate system. Then, when converting the three-dimensional world coordinates of the center point of the window opening area to the depth camera to the coordinate value under the robotic arm coordinate system, only the displacement changes on the z_robot_arm axis and y_robot_arm axis under the robotic arm coordinate system need to be considered. Moreover, the robot arm starts to move from its original position to drive the inspection equipment on it to the window opening area, and then returns to the initial position after inspection. The depth camera is installed on the support rod on the safety island. After the depth camera is installed and debugged, the angle θ between the optical axis of the depth camera and the horizontal line pointing to the vehicle and parallel to the ground can be measured. The height difference between the origin of the robotic arm coordinate system and the origin of the depth camera coordinate system in the direction perpendicular to the ground is , the front-to-back position difference between the origin of the robot coordinate system and the origin of the depth camera coordinate system in the direction parallel to the ground is .
[0138] The specific formula for converting the three-dimensional world coordinates of the center point of the window area to the depth camera into the coordinate values in the robotic arm coordinate system is as follows:
[0139]
[0140] in, is the coordinate value of the center point of the window opening area on the y_robot_arm axis of the robot arm coordinate system, is the coordinate value of the center point of the window area on the x_robot_arm axis of the robot coordinate system. The movement drives the inspection device to extend to the window area to automatically inspect the situation inside the car. The inspection device can be, but is not limited to, a surveillance camera with a 360-degree viewing angle and / or an infrared detector and other equipment.
[0141] The above-mentioned in-vehicle inspection system can automatically inspect the internal conditions of passing vehicles, and can realize unattended inspection functions on the lanes. Inspectors can view the inspection results in the background at a distance.
[0142] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the protection scope of the present application.
Claims
1. A vehicle window positioning detection method, characterized in that: include: Acquire image data F of the side of the vehicle collected by a depth camera, wherein the image data F includes a color image F1 and a depth image F2; Detecting the degree of opening Deg of the vehicle window when it is open based on the color image F1 or detecting the degree of opening Deg of the vehicle window when it is open based on the image data F; When the opening degree is greater than the first threshold, a histogram of depth information is performed on the depth image F2 to obtain a distance D from the vehicle body to the depth camera; and The three-dimensional world coordinate value from the center point of the window area to the depth camera is calculated based on the distance D from the vehicle body to the depth camera; Among them, the depth information of the depth image F2 is counted by a histogram to obtain the distance D from the vehicle body to the depth camera, including: The threshold range of depth information for histogram statistics is set according to the position of the depth camera installed on the lane and the width of the lane, where H1<H2; Perform a histogram statistics on the number of pixels corresponding to each depth information value within the threshold range [H1, H2] in the depth image F2; The depth information value corresponding to the maximum value of the vertical axis of the histogram is taken as the distance D from the vehicle body to the depth camera.
2. The vehicle window positioning detection method according to claim 1, characterized in that: Detecting the degree of opening and closing of the window Deg based on the color image F1 includes: Based on the first target detection model, the middle window target Mid closest to the center of the color image F1 is detected among all the windows in the color image F1 box ; Based on the second target detection model, the glass targets of all windows in the color image F1 are detected to obtain the glass target set G glass ; In the glass target set G glass Find the target Mid that meets the first preset condition box Glass Target Mid glass ; If, in the glass target set G glass Failed to find a glass target frame that meets the first preset condition Mid glass ;Then the window opening degree Deg=100%; Otherwise, according to the middle window target Mid box and glass target Mid glass Calculate the window opening degree Deg: window opening degree Deg = 1-(H Midglass / H Midbox ); Among them, H Midbox For the window target Mid box The height of the target box, H Midglass For glass target Mid glass The height of the target box.
3. The vehicle window positioning detection method according to claim 1, characterized in that: The opening degree Deg of the window when the vehicle window is opened is detected based on the image data F, including: Based on the first target detection model, the middle window target Mid closest to the center of the color image F1 is detected among all the windows in the color image F1 box ; Perform histogram statistics of the depth information of the depth image F2 to obtain the distance D from the vehicle body to the depth camera; The depth image F2 is binarized with the distance D as a second threshold to obtain a binarized image; Creating a first structure element and a second structure element for an image erosion operation, wherein the size of the first structure element is larger than the second structure element; Sequentially using the first structuring element and the second structuring element to perform an erosion operation on the binary image to obtain a first image; Performing a connected domain solving operation on the first image to obtain a second image including a plurality of connected domains; Calculate the minimum bounding rectangle of all connected domains, and select the minimum bounding rectangle whose width meets the second preset condition from all the minimum bounding rectangles to form a minimum bounding rectangle set G Rect ; In the minimum bounding rectangle set G Rect Select the smallest bounding rectangle Mid that is closest to the center area of the depth image F2 Rect ; According to the middle window target Mid box and the minimum enclosing rectangle Mid Rect Calculate the window opening degree Deg: window opening degree Deg = H grect / H Midbox ; Among them, H Midbox For the window target Mid box Height, H grect is the minimum enclosing rectangle Mid Rect height.
4. The vehicle window positioning detection method according to claim 2 or 3, characterized in that: Based on the first target detection model, the middle window target Mid closest to the center of the color image F1 is detected among all the windows in the color image F1 box ,include: Detect all vehicle window target frames in the color image F1 based on the first target detection model; Calculate the area of all window targets, sort all area values from large to small, and select the first n window target frames after sorting to form the window target set G box ; Calculate the window target set G box The Euclidean distance between the center point of each window target frame in and the center point of the color image F1; Select the window target corresponding to the smallest Euclidean distance value as the middle window target Mid box .
5. The vehicle window positioning detection method according to claim 2, characterized in that: Set the middle window target to Mid box The horizontal coordinates of the upper left corner and the lower right corner of the target box are and , assuming that the glass target set G glass A glass target glass i The horizontal coordinates of the upper left corner and the lower right corner of the target box are and ; Mid window target box and glass target glass i When the following first preset condition is met, the glass target set G glass A glass target glass i To find the glass target Mid glass ; Among them, the first preset condition is: and The value of is greater than a third threshold; in, To obtain the maximum value, To perform the minimum value operation.
6. The vehicle window positioning detection method according to claim 3, characterized in that: Any minimum enclosing rectangle R ecti Belongs to the set G Rect The second precondition that needs to be met is: W Recti > T W and W Recti < 0.8·W F2 Among them, W Recti is the minimum external target box R ecti Width, W F2 is the width of the depth image F2, T W A fourth threshold value regarding width is set.
7. The vehicle window positioning detection method according to claim 4, characterized in that: When the opening degree = 100%, the 3D world coordinate value from the center point of the window area to the depth camera is calculated based on the distance D from the vehicle body to the depth camera, which is: ; ; in, , , and The intrinsic parameter matrix K from the depth camera; , are the horizontal and vertical coordinates of the origin of the image physical coordinate system in the image pixel coordinate system respectively; , is the horizontal and vertical coordinates of the center point of the window area in the image, that is, the obtained middle window target Mid box The horizontal and vertical coordinates of the center point in the image; and are the focal length of the depth camera in the horizontal and vertical directions respectively; It is the 3D world coordinate value from the center point of the windowing area to the depth camera.
8. The vehicle window positioning detection method according to claim 3, characterized in that: When the opening degree is ≠100%, the 3D world coordinate value from the center point of the window area to the depth camera is calculated based on the distance D from the vehicle body to the depth camera, which is: ; ; in, , , and The intrinsic parameter matrix K from the depth camera; , are the horizontal and vertical coordinates of the origin of the image physical coordinate system in the image pixel coordinate system respectively; , is the horizontal and vertical coordinates of the center point of the window opening area in the image. The center point of the window opening area is the minimum circumscribed rectangle R ecti The horizontal and vertical coordinates of the center point in the image; and are the focal length of the depth camera in the horizontal and vertical directions respectively; It is the 3D world coordinate value from the center point of the windowing area to the depth camera.
9. An in-vehicle inspection system, characterized in that: include: A depth camera disposed on a safety island on one side of the lane is used to collect a side image of a vehicle parked in a preset position of the lane to obtain image data F, wherein the image data F includes a color image F1 and a depth image F2; A mechanical arm disposed on the same side of the lane as the depth camera can move forward and backward relative to the vehicle on the lane and can move up and down relative to the ground; An inspection device disposed on the mechanical arm; A first processing module, communicatively connected to the depth camera, for executing the steps of the vehicle window positioning detection method according to any one of claims 1 to 8 to obtain a three-dimensional world coordinate value from the center point of the window opening area to the depth camera; and The second processing module is communicatively connected to the first processing module and is used to convert the three-dimensional world coordinates from the center point of the window area to the depth camera obtained by the first processing module into coordinate values in the robotic arm coordinate system.
Citation Information
Patent Citations
Target detection and visual positioning method based on YOLO series
CN112927297A
Vehicle control method, device and equipment and storage medium
CN115478757A
Vehicle window control method and device, vehicle and storage medium
CN116181183A
Intelligent vehicle window control method and device, electronic equipment and storage medium
CN117661960A
In-vehicle inspection method and system based on intelligent vision
CN118038423A
