Real-time vehicle shape analysis device and method based on vision sensor for detecting improper loading

Abstract

A vision sensor-based real-time vehicle shape analysis device and method for monitoring improper loading are disclosed. The vision sensor-based real-time vehicle shape analysis device for monitoring improper loading includes a vision sensor positioned toward a road surface to acquire data for calculating vehicle specifications, and a processor that analyzes the vehicle shape based on the data acquired from the vision sensor and calculates the vehicle specifications based on the analysis results. Accordingly, by analyzing the vehicle shape in real time based on data acquired through a stereo vision sensor, it is possible to accurately calculate distance information between the vehicle and the sensor without expensive equipment.

Description

Technology Field

[0001] The present invention relates to a vision sensor-based real-time vehicle shape analysis device and method for detecting improper loading, and more specifically, to a vision sensor-based real-time vehicle shape analysis device and method for detecting improper loading that analyzes the shape of a moving vehicle in real time using data acquired from a vision sensor and calculates vehicle specification information based on the analysis results to determine whether there is improper loading. Background Technology

[0003] Conventionally, laser scanner (LiDAR)-based devices or single-camera image analysis technology have been primarily used to measure vehicle specifications.

[0004] LiDAR-based devices can obtain relatively accurate distance information by directly scanning the outer shape of a vehicle using laser pulses; however, they have limitations in that they are difficult to apply to a wide range of road environments due to high equipment costs and significant installation and maintenance expenses. Additionally, these devices are only capable of measuring vehicles passing through specific enforcement points, making it difficult to measure specifications in real time for high-speed vehicles or vehicles moving simultaneously across multiple lanes.

[0005] Meanwhile, while single-camera-based image analysis technology can estimate the size or contours of a vehicle within an image, it cannot directly acquire depth information using only monocular images. Consequently, measurement accuracy drops significantly depending on changes in the distance between the vehicle and the camera, deviations in the shooting angle, and variations in illumination. In particular, in real-time shooting environments, the technology is susceptible to external factors such as brightness differences between frames, shadows, and road reflections, making it difficult to reliably separate vehicle shapes or calculate reliable specifications.

[0006] Accordingly, there has been an increased need for technology that is cost-effective, easy to install, and capable of reliably analyzing vehicle shapes in real time to precisely calculate distance information between the vehicle and the vision sensor. In addition, in order to effectively crack down on overloaded vehicles that exceed the vehicle specification standards stipulated by road laws, it is necessary to accurately measure the height, length, and width of moving vehicles; therefore, there has been an increased need for vision sensor-based real-time vehicle shape analysis technology that can deliver stable performance under various road brightness conditions and vehicle movement environments. Prior art literature

[0008] Korean Registered Patent No. 10-2816936 (Published June 5, 2015) The problem to be solved

[0009] The objective of the present invention is to provide a vision sensor-based real-time vehicle shape analysis device and method for detecting loading defects, which analyzes the shape of a moving vehicle in real time using data acquired from a vision sensor and calculates vehicle specification information based on the analysis results to determine whether there is a loading defect.

[0010] Other objects and advantages of the present invention may be understood from the following description and will be more clearly understood by one embodiment. Furthermore, it will be readily apparent that the objects and advantages of the present invention can be realized by the means and combinations thereof set forth in the claims. means of solving the problem

[0012] A vision sensor-based real-time vehicle shape analysis device for detecting loading defects according to one embodiment of the present invention for achieving such an objective comprises: a vision sensor positioned to face a road surface to acquire data for calculating vehicle specifications; and a processor that analyzes the shape of a vehicle based on data acquired from the vision sensor and calculates vehicle specifications based on the analysis result.

[0013] The above processor is characterized by generating depth information by calculating a time difference from the above data.

[0014] The above processor is characterized by analyzing the road surface brightness value of a frame in which the vehicle is not present to adjust the exposure time, and using this to update the background image from the frame.

[0015] The processor is characterized by analyzing the depth difference between the depth image of the current frame and the background image to determine whether a vehicle has entered, and generating a mask image of the vehicle with pixels greater than or equal to a preset height relative to the ground.

[0016] The processor is characterized by sequentially collecting frames to which the mask image is applied as segmented images while the vehicle is present, and terminating the collection of segmented images when the vehicle leaves.

[0017] The processor is characterized by detecting edges for a vehicle region of the segmented image and enhancing the contour of the vehicle using at least one of bidirectional filtering and morphological operation.

[0018] The processor is characterized by analyzing the brightness of the edge pixels as a histogram to remove edges that are not valid for stitching.

[0019] The processor is characterized by calculating a matching score by combining the matching rate and the inverse of the average brightness difference in the overlapping area between the segmented images, and stitching the segmented images to the position where the matching score is maximized.

[0020] The above processor is characterized by calculating distance information between the vision sensor and the vehicle using a triangulation method based on the parallax information of the stitched image and / or a focal length and / or baseline.

[0021] The processor is characterized by calculating the specifications of a vehicle based on the distance information and determining the vehicle as an improperly loaded vehicle if the calculated specifications of the vehicle exceed a preset standard.

[0022] A method for analyzing a vehicle shape in a vision sensor-based real-time vehicle shape analysis device for detecting loading defects according to another embodiment of the present invention comprises: a step of receiving data for calculating the specifications of a vehicle; and a step of analyzing the shape of a vehicle based on the data for calculating the specifications of a vehicle, and calculating the specifications of a vehicle based on the analysis result.

[0023] According to another embodiment, a computer-readable recording medium may include a program for executing a vehicle shape analysis method on a computer. Effects of the invention

[0025] According to various embodiments of the present invention as described above, by analyzing the vehicle shape in real time based on data acquired through a stereo vision sensor, it is possible to accurately calculate distance information between the vehicle and the sensor without expensive equipment.

[0026] In addition, by applying exposure control and background update technologies, it is effective in providing stable analysis performance even in environments of varying brightness.

[0027] In addition, the shape of a high-speed vehicle can be reliably reconstructed using a segmented image-based stitching algorithm. Based on this, specifications such as the height, length, and width of the vehicle can be precisely measured, and vehicles with improper loading exceeding specification standards can be quickly identified. Accordingly, this has the effect of contributing to improved road safety and increased efficiency in enforcement operations.

[0028] The effects obtainable from the disclosed embodiments are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art to which the disclosed embodiments belong from the description below. Brief explanation of the drawing

[0030] FIG. 1 is a diagram illustrating the configuration of a vision sensor-based real-time vehicle shape analysis device for detecting loading defects according to an embodiment of the present invention. FIGS. 2 to 12 are drawings illustrating a vision sensor-based real-time vehicle shape analysis device for detecting loading defects according to an embodiment of the present invention. FIG. 13 is a flowchart illustrating a vehicle shape analysis method according to one embodiment of the present invention. FIG. 14 is a block diagram showing the specific configuration of a vision sensor-based real-time vehicle shape analysis device for detecting loading defects illustrated in FIG. 1 according to an embodiment of the present invention. FIG. 15 is a drawing relating to a software module stored in a storage unit according to an embodiment of the present invention. Specific details for implementing the invention

[0031] Various embodiments are described in detail below with reference to the attached drawings. The embodiments described below may be implemented in various different forms. In order to explain the features of the embodiments more clearly, detailed descriptions of matters widely known to those skilled in the art to which the following embodiments belong have been omitted. Additionally, parts of the drawings unrelated to the description of the embodiments have been omitted, and similar parts throughout the specification have been given similar reference numerals.

[0032] Throughout the specification, when a configuration is described as being "connected" to another configuration, this includes not only cases where they are "directly connected," but also cases where they are "connected with another configuration in between." Furthermore, when a configuration is described as "including" another configuration, this means that, unless specifically stated otherwise, it does not exclude other configurations but may include additional configurations.

[0033] Meanwhile, in describing the present invention with reference to the drawings, detailed descriptions of related known functions or configurations are omitted if it is determined that such detailed descriptions could unnecessarily obscure the essence of the invention. Furthermore, the terms described below are defined considering their functions in the present invention, and these may vary depending on the intentions or relationships of the user or operator. Therefore, their definitions should be based on the content throughout this specification.

[0034] FIG. 1 is a diagram illustrating the configuration of a vision sensor-based real-time vehicle shape analysis device for detecting improper loading according to an embodiment of the present invention, and FIGS. 2 to 12 are diagrams for explaining a vision sensor-based real-time vehicle shape analysis device for detecting improper loading according to an embodiment of the present invention.

[0035] Referring to FIG. 1, a vision sensor-based real-time vehicle shape analysis device (100) for detecting loading defects according to the present embodiment includes a vision sensor (110) and a processor (120).

[0036] A vision sensor (110) can acquire data for calculating the specifications of a vehicle. More specifically, the vision sensor (110) may be positioned to face the road surface to calculate the specifications of a vehicle. Here, the vision sensor (110) may be a stereo vision camera. According to an embodiment, there may be two vision sensors (110) and they may be positioned to face the road surface vertically to calculate the specifications of a vehicle. That is, the vision sensor (110) according to the present embodiment may include a pair of stereo vision cameras positioned on the left and / or right, which are positioned to face the road surface vertically to calculate the specifications of a vehicle.

[0037] The processor (120) can control the overall operation of the vision sensor-based real-time vehicle shape analysis device (100) for detecting loading defects and may be any one of a CPU, GPU, or Arduino. That is, the processor (120) can control other components included in the vision sensor-based real-time vehicle shape analysis device (100) for detecting loading defects that perform the vision sensor-based real-time vehicle shape analysis operation for detecting loading defects. In addition, the processor (120) may execute a program stored in the storage unit (130) described later, read a file stored in the storage unit (130), or store a new file in the storage unit (130).

[0038] The processor (120) can analyze the shape of the vehicle based on data acquired from the vision sensor (110) and calculate the specifications of the vehicle based on the analysis results. At this time, the data acquired from the vision sensor (110) may include image data acquired to calculate the specifications of the vehicle, and the image data may include a depth map generated by extracting depth information based on disparity information, focal length, and baseline calculated from left and right cameras. Meanwhile, the data acquired from the vision sensor (110) may be vehicle shape data, i.e., vehicle specification data, for calculating the specifications of the vehicle. At this time, the specifications of the vehicle may be the height, length, width, etc. of the vehicle. Accordingly, the processor (120) can calculate the specifications of the vehicle, i.e., height, length, width, etc., by processing the information included in the image data acquired from the vision sensor (110).

[0039] The processor (120) can generate depth information by calculating the time difference from the data. At this time, the depth information may mean the distance value from the vision sensor (110) to the vehicle, i.e., distance information.

[0040] More specifically, the processor (120) can generate depth information by calculating the disparity between the image data acquired through the left vision sensor and the image data acquired through the right vision sensor among the data.

[0041] According to an embodiment, the processor (120) can calculate the parallax between the two image data described above and calculate the depth information of each pixel, i.e., the distance information from the vision sensor to the surface of the vehicle, using a known triangulation method based on the parallax, focal length, and baseline distance.

[0042] The processor (120) can analyze the road surface brightness value of a frame in which no vehicle is present to adjust the exposure time, and use this to update the background image from the frame.

[0043] For example, the processor (120) can generate a stable background image by analyzing the brightness value of the road surface in a frame where no vehicle is present and automatically adjusting the exposure time. More specifically, the processor (120) can set the center of the current frame and / or a predefined road surface reference area as a Region of Interest (ROI), and calculate the average and / or median of the brightness values ​​of the pixels within the ROI.

[0044] The processor (120) can control the exposure time by shortening it when the calculated brightness value is higher than a preset target brightness range, and conversely, by increasing it when the brightness value is lower than the target range. At this time, the processor (120) can limit the amount of change between frames by setting upper and lower limits on the exposure time adjustment value to prevent screen flickering caused by excessive exposure changes. Meanwhile, the adjusted exposure time can be reflected immediately when the next frame is acquired, and the processor (120) can update the frame acquired under the adjusted exposure time as a background image in a vehicle absence state. The background image updated in this way can be used as a reference image for the vehicle entry determination and vehicle mask generation steps.

[0045] Additionally, to minimize brightness fluctuations between frames after the point in time when it is determined that a vehicle has entered, the processor (120) can maintain a fixed exposure time and control automatic exposure control based on road surface brightness analysis to resume when the vehicle has completely exited the frame.

[0046] The processor (120) can determine whether a vehicle has entered by analyzing the depth difference between the depth image of the current frame and the background image, and can generate a mask image of the vehicle with pixels greater than or equal to a preset height relative to the ground.

[0047] For example, the processor (120) can determine whether a vehicle has entered by comparing the depth image obtained in the current frame with the depth value of the background image updated in a state where no vehicle is present. More specifically, the processor (120) calculates the difference from the background depth value for each pixel of the current frame, and if the number of pixels where the difference value exceeds a preset depth change threshold is greater than or equal to a predetermined reference number of pixels, it can determine that a vehicle has entered in that frame.

[0048] When vehicle entry is confirmed, the processor (120) may generate a vehicle mask image by considering only pixels above a preset height relative to the ground among the areas where depth differences occur as vehicle areas. Here, the height relative to the ground may be preset according to the installation environment and / or false detection prevention conditions and may be used to remove fine irregularities of the road surface, vehicle undercarriage shadows and / or road surface noise.

[0049] The processor (120) can exclude all pixels with a depth value less than the ground reference height to minimize false detections caused by shadows, reflections, and road surface patterns on the underside of the vehicle, and can generate a binary vehicle mask image containing only pixels that satisfy the above-described conditions. The generated vehicle mask can subsequently be used as a standard for the frame segmentation image collection and stitching area extraction steps.

[0050] The processor (120) may also exclude the vehicle's lower shadow and / or road noise by removing pixels below a preset height relative to the ground.

[0051] The processor (120) can sequentially collect frames with mask images applied as segmented images while the vehicle is present, and can stop collecting segmented images when the vehicle leaves.

[0052] For example, the processor (120) can sequentially collect the current frame with the applied mask image as segmented images from the time the vehicle entry is confirmed until the vehicle completely exits the frame. More specifically, the processor (120) can determine that the frame is a vehicle presence section if the vehicle mask generated in each frame maintains a size greater than a certain size, and can store the original image, depth image, and vehicle mask of the frame as a single segmented image in a separate storage space (not shown).

[0053] At this time, the processor (120) can collect all frames without omission, even if the vehicle mask among consecutive frames has almost the same pattern, thereby ensuring sufficient front, rear, and side contour information of the vehicle required in the subsequent stitching step. On the other hand, if the vehicle mask of the current frame decreases to less than a certain threshold number of pixels or if a depth difference no longer occurs, it is determined that the vehicle is out of sight, and the processor (120) can immediately terminate the segmented image collection step.

[0054] The multiple segmented images collected in this way can be used for edge analysis and / or stitching processing in subsequent steps and can be utilized as basic data for restoring the overall shape of the vehicle and calculating specifications.

[0055] The processor (120) can detect edges for vehicle regions of a segmented image and enhance the contour of the vehicle using at least one of bilateral filtering and morphology operations. In this case, the morphology operation may be either a gradient and / or a top-hat.

[0056] For example, the processor (120) can detect edges in a region designated as a vehicle mask in a segmented image and enhance the contour of the vehicle based on the edge information. More specifically, the processor (120) can extract edges of the vehicle region using a Canny operator and / or a Sobel operator, and then apply bidirectional filtering to remove pixels with weak intensity of detected edge values ​​to reduce noise. At this time, the processor (120) can set spatial weight and brightness weight parameters of the bidirectional filtering to maintain boundaries by considering the brightness difference between neighboring pixels.

[0057] Additionally, the processor (120) may perform at least one of a gradient operation and / or a top-hat operation during a morphological operation to further enhance the edge enhancement effect. The morphological gradient operation emphasizes the contrast of the boundary line by utilizing the difference between the dilation image and the erosion image, and the top-hat operation can express the outer contour of the vehicle more clearly by highlighting bright structures.

[0058] Finally, the processor (120) can generate an improved edge map through bidirectional filtering and morphological operations, and use the edge map as a matching criterion for a subsequent stitching step, thereby enabling accurate extraction of the vehicle shape.

[0059] The processor (120) can analyze the brightness of edge pixels as a histogram to remove edges that are not valid for stitching. In this case, according to the embodiment, stitching is a process of reconstructing a part of the shape of the vehicle observed in individual frames into a continuous and integrated single image by matching feature information of each image based on an overlapping area between a plurality of segmented images acquired while the vehicle is moving. In the present invention, this may refer to an algorithmic technology that determines the optimal matching position between frames using edge-based feature analysis, histogram-based edge filtering, and calculation of a matching score based on a matching rate and brightness difference, thereby accurately restoring the entire contour and depth information of the vehicle.

[0060] For example, the processor (120) can collect the brightness (gradient magnitude or intensity) of each pixel for edge pixels detected in the vehicle area of ​​the segmented image and analyze the distribution in the form of a histogram. More specifically, the processor (120) can calculate the brightness distribution by accumulating the brightness values ​​of the edge pixels by bins and determine that pixels in the bins with low brightness values ​​in the distribution described above correspond to noise that impairs matching accuracy during the stitching process.

[0061] Based on the results of histogram analysis, the processor (120) can classify edge pixels whose brightness values ​​fall within a preset threshold ratio (e.g., less than a certain percentage of the upper brightness values) as "invalid edges" and remove the pixels or replace them with a value of 0 to exclude them from the final edge map. Through this process, pixels with low edge values ​​that have little matching contribution and / or irregular edges caused by vehicle surface reflections, road surface patterns, etc., can be effectively removed.

[0062] Additionally, the processor (120) can reduce the amount of matching computation by reducing the number of edges used as comparison targets in the stitching algorithm by performing histogram-based filtering, and improve the quality of the edges to improve the matching rate between frames and prevent the selection of incorrect stitching locations.

[0063] Additionally, the processor (120) can fix the exposure time after the point in time when vehicle entry is determined to be, thereby minimizing brightness fluctuations between frames. Accordingly, the performance of stitching can be improved.

[0064] The processor (120) can calculate a matching score by combining the matching rate and the inverse of the average brightness difference in the overlapping area between the segmented images, and stitch the segmented images to the position where the matching score is maximized.

[0065] Meanwhile, the matching rate according to the present embodiment is calculated through the following mathematical formula 1, the average brightness difference is calculated through the following mathematical formula 2, and the matching score can be calculated through the following mathematical formula 3.

[0066] [Mathematical Formula 1]

[0067]

[0068] [Mathematical Formula 2]

[0069]

[0070] [Mathematical Formula 3]

[0071]

[0072] For example, the processor (120) can calculate a matching score by calculating a matching rate and an average brightness difference based on the overlap area between the segmented images, and determine the position where the matching score is maximum as the stitching position. More specifically, the processor (120) can calculate the number of edge pixels that can be matched between the two images by moving the edge image stitched in the previous frame and the edge image of the current frame by a predetermined range in the x-axis and y-axis directions, and calculate the ratio as the matching rate.

[0073] Additionally, the processor (120) may calculate the average brightness difference by summing the brightness value differences of the matched edge pixels within the overlapping area and dividing the result by the number of matched pixels, and use the reciprocal of the average brightness difference as a matching evaluation metric. Afterward, the processor (120) may calculate a matching score by combining the calculated matching rate with the reciprocal of the average brightness difference, and determine the coordinate with the largest matching score as the optimal stitching position of the current frame.

[0074] In this process, if there are multiple locations with the same matching score, the processor (120) can prevent incorrect alignment by prioritizing the location with a larger number of matched pixels and selecting it as the stitching location. Meanwhile, the determined stitching location can be reflected in the subsequent generation of edge stitching images and / or depth stitching images to improve the continuous restoration accuracy of the vehicle shape.

[0075] Additionally, the processor (120) may select the location with the more matched pixels as the final stitching location if the matching scores are the same.

[0076] For example, the processor (120) can determine the final stitching location by comparing the number of edge pixels matched at each candidate location when the same matching score is calculated at two or more different candidate locations as a result of calculating the matching score between the segmented images. More specifically, the processor (120) can calculate the number of edge pixels that actually correspond within the overlapping area for candidate locations with the same matching score and select the location with the highest number of matching pixels as the final stitching location. This method has the effect of preventing vehicle shape distortion due to incorrect alignment and improving the stability of the stitching result by determining the stitching location based on the reliability of the actual matching information in a competitive situation between two candidate locations that is difficult to distinguish by the matching score alone.

[0077] The processor (120) can calculate distance information between the vision sensor (110) and the vehicle using a triangulation method based on the parallax information of the stitched image and / or the focal length and / or baseline.

[0078] For example, the processor (120) can calculate a disparity value from the edge stitching image and depth stitching image of the left and right vision sensors (110) obtained through stitching, and calculate distance information between the vision sensors (110) and the vehicle by applying the disparity information to a known triangulation method. More specifically, the processor (120) can define the difference between the x-axis position of a specific pixel detected in the left stereo image and the x-axis position of an identical object appearing in the right image as a disparity value, and calculate the depth value (Z) of the corresponding pixel using a known triangulation method based on a pre-corrected focal length (f) and the distance between the centers of the two camera lenses (baseline, b).

[0079] The processor (120) generates a distance map between the vehicle and the vision sensor (110) by repeatedly applying the calculated depth value to each corresponding pixel of the stitched overall vehicle shape image, and can utilize this distance data in the step of calculating the height, length, and width specifications of the vehicle. At this time, the processor (120) can improve the accuracy of the calculation by converting the depth value into a distance in mm units using pixel-per-millimeter information obtained during the initial calibration process.

[0080] The processor (120) calculates the specifications of the vehicle based on distance information, and if the calculated specifications of the vehicle exceed a preset standard, the vehicle can be determined to be a vehicle with improper loading.

[0081] For example, the processor (120) can calculate specifications such as the height, length, and width of the vehicle using the calculated distance information. More specifically, the processor (120) can convert the depth value (Z) of each pixel in the stitched depth image into a distance in mm units, and then extract the maximum values ​​of the coordinates corresponding to the height direction (Z-axis), length direction (X-axis), and width direction (Y-axis) within the vehicle mask area to determine the vehicle specification values.

[0082] The processor (120) compares the calculated specification value with pre-stored allowable standard specifications (e.g., maximum height, maximum load length, maximum load width, etc.) according to the Road Traffic Act and / or system setting values, and if the calculated specification value exceeds the standard specifications in any direction, the corresponding vehicle may be determined to be a vehicle with improper loading. Additionally, the processor (120) may provide basis data for determining improper loading by calculating the items exceeding the standard and the amount exceeding the standard together and transmitting them to an enforcement system and / or a separate server above.

[0083] With this configuration, accurate specifications can be calculated for a moving vehicle in real time, and by automatically determining whether the specifications are exceeded, the efficiency of enforcement against improper loading can be improved.

[0084] Meanwhile, FIGS. 2 to 12 are drawings for explaining a vision sensor-based real-time vehicle shape analysis device (100) for detecting loading defects according to an embodiment of the present invention.

[0085] First, FIG. 2 shows a real-time monitoring view for detecting a moving vehicle equipped with a vision sensor (110).

[0086] FIG. 2(a) shows image data (image) captured when no vehicle passes while the vision sensor (110) is installed above the road, and FIG. 2(b) shows image data (image) captured when a vehicle passes. At this time, the vision sensor, i.e., the left and right cameras, operate in real-time at 30fps and provide a reference frame (image data (image)) to check whether a vehicle has entered by vertically capturing the road surface from a preset height, for example, 6m. Meanwhile, although the vision sensor, i.e., the left and right cameras are described as operating at 30fps in this embodiment, they are not limited thereto and can acquire image data (image) by operating at various speeds according to the user's settings.

[0087] Figure 3 illustrates a process of automatically adjusting the exposure time by analyzing the brightness of the road surface in a frame where no vehicle is present, and updating the background image based on this.

[0088] More specifically, FIG. 3(a) shows an example of ROI settings for road surface brightness analysis, FIG. 3(b) shows an exposure time adjustment code based on the average ROI brightness value, and FIG. 3(c) shows a background image screen that is updated in real time.

[0089] Figure 4 illustrates a procedure for determining whether a vehicle has entered by using the depth difference between the background image and the current frame.

[0090] More specifically, FIG. 4(a) shows the latest background depth image, FIG. 4(b) shows the current frame at the time of vehicle entry, FIG. 4(c) shows the result of detecting the presence of a vehicle through the depth difference, and FIG. 4(d) shows a vehicle mask image generated by applying a ground-based height threshold.

[0091] Figure 5 shows segmented images collected from the right camera while the vehicle is in the field of view and the preprocessing results for the images.

[0092] More specifically, FIG. 5(a) shows the original frame, FIG. 5(b) shows the depth image, and FIG. 5(c) shows the vehicle area mask image.

[0093] Figure 6 shows the edge detection and contour enhancement process performed on the vehicle mask area.

[0094] More specifically, FIG. 6(a) shows the original frame image, FIG. 6(b) shows the edge detection result, FIG. 6(c) shows the result of removing weak edges through bidirectional filtering, and FIG. 6(d) shows the result of emphasizing the vehicle contour by applying morphological gradient and top-hat operations.

[0095] Figure 7 shows the result of removing weak edges that are not valid for stitching through histogram analysis of the brightness values ​​of edge pixels. With this process, only edges that contribute significantly to stitching can be retained.

[0096] Figure 8 shows a process of calculating a matching score using the matching rate and average brightness difference in the overlapping area between segmented images, and determining the stitching position based on this.

[0097] More specifically, FIG. 8(a) shows the previously stitched edge image, FIG. 8(b) shows the edge image of the current frame, and FIG. 8(c) shows the stitching result according to the calculated matching score.

[0098] Figure 9 is the final result of performing matching score-based stitching on segmented images collected from the right camera.

[0099] More specifically, FIG. 9(a) shows an edge stitching image and FIG. 9(b) shows a depth stitching image with the same stitching coordinates applied. This depth stitching result can be used to calculate vehicle specifications.

[0100] Figure 10 is a diagram illustrating the principle of stereo vision triangulation, which calculates the depth (Z) to an object by utilizing the disparity, focal length (f), and baseline (b) of the left and right cameras.

[0101] Figure 11 is a drawing showing the height, length, and width specifications of a vehicle calculated based on stitched depth data, along with the final stitched image and a 3D visualization example.

[0102] FIG. 12 is a flowchart showing the overall flow of the vehicle shape analysis algorithm of the present invention in steps, and may be composed of a background updater, vehicle entry determination, segmented image collection, edge analysis, stitching, and distance and specification calculation.

[0103] Meanwhile, the vision sensor-based real-time vehicle shape analysis device (100) for detecting loading defects according to the present embodiment can be installed on a structure of a specific height.

[0104] In addition, the vision sensor-based real-time vehicle shape analysis device (100) for detecting loading defects according to the present embodiment may include a communication unit (not shown).

[0105] A communication unit (not shown) can perform wired or wireless communication with other devices and / or networks. To this end, the communication unit (not shown) may include a communication module that supports at least one of various wired or wireless communication methods. For example, the communication module may be implemented in the form of a chipset.

[0106] Meanwhile, wireless communication supported by the communication unit (not shown) may be, for example, wireless mobile communication such as Wi-Fi (Wireless Fidelity), Wi-Fi Direct, Bluetooth, Bluetooth Low Energy (BLE), Ultra Wide Band (UWB), Near Field Communication (NFC), LTE, and LTE-Advanced. Additionally, wired communication supported by the communication unit (not shown) may be, for example, USB or HDMI (High Definition Multimedia Interface).

[0107] As described above, the present invention has the effect of accurately calculating distance information between a vehicle and a sensor without expensive equipment by analyzing the vehicle shape in real time based on data acquired through a stereo vision sensor.

[0108] In addition, by applying exposure control and background update technologies, it is effective in providing stable analysis performance even in environments of varying brightness.

[0109] In addition, the shape of a high-speed vehicle can be reliably reconstructed using a segmented image-based stitching algorithm. Based on this, specifications such as the height, length, and width of the vehicle can be precisely measured, and vehicles with improper loading exceeding specification standards can be quickly identified. Accordingly, this has the effect of contributing to improved road safety and increased efficiency in enforcement operations.

[0111] FIG. 13 is a flowchart illustrating a vehicle shape analysis method according to one embodiment of the present invention.

[0112] Referring to FIG. 13, a vehicle shape analysis method according to one embodiment of the present invention includes a data receiving step (S1210) and a vehicle shape analysis step (S1220).

[0113] In step S1210, a vision sensor-based real-time vehicle shape analysis device (100) for detecting loading defects can receive data for calculating vehicle specifications obtained from a vision sensor (110).

[0114] In step S1220, a vision sensor-based real-time vehicle shape analysis device (100) for detecting improper loading can analyze the shape of a vehicle based on data acquired from a vision sensor (110) and calculate the specifications of the vehicle based on the analysis results. At this time, the data acquired from the vision sensor (110) may include image data acquired to calculate the specifications of the vehicle, and the image data may include a depth map generated by extracting depth information based on disparity information, focal length, and baseline calculated from left and right cameras.

[0115] A vision sensor-based real-time vehicle shape analysis device (100) for detecting improper loading can generate depth information by calculating a time difference from data. At this time, the depth information may refer to the distance value from the vision sensor (110) to the vehicle, i.e., distance information.

[0116] A vision sensor-based real-time vehicle shape analysis device (100) for detecting improper loading analyzes the brightness value of the road surface of a frame where no vehicle is present to adjust the exposure time, and can use this to update the background image from the frame.

[0117] A vision sensor-based real-time vehicle shape analysis device (100) for detecting improper loading can determine whether a vehicle has entered by analyzing the depth difference between the depth image of the current frame and the background image, and can generate a mask image of the vehicle with pixels greater than or equal to a preset height relative to the ground.

[0118] A vision sensor-based real-time vehicle shape analysis device (100) for detecting improper loading can sequentially collect frames with applied mask images as segmented images while a vehicle is present, and can stop collecting segmented images when the vehicle leaves.

[0119] A vision sensor-based real-time vehicle shape analysis device (100) for detecting improper loading can detect edges on vehicle regions of segmented images and enhance the contour of the vehicle using at least one of bidirectional filtering and morphology operation. At this time, the morphology operation may be either a gradient and / or a top-hat.

[0120] A vision sensor-based real-time vehicle shape analysis device (100) for detecting improper loading can remove edges that are not valid for stitching by analyzing the brightness of edge pixels as a histogram. In this case, according to the embodiment, stitching is a process of reconstructing a part of the vehicle shape observed in individual frames into a continuous and integrated single image by matching feature information of each image based on an overlapping area between multiple segmented images acquired while the vehicle is moving. In the present invention, this may refer to an algorithm technology that determines the optimal matching position between frames using edge-based feature analysis, histogram-based edge filtering, and calculation of a matching score based on a matching rate and brightness difference, thereby accurately restoring the overall contour and depth information of the vehicle.

[0121] A vision sensor-based real-time vehicle shape analysis device (100) for detecting improper loading can calculate a matching score by combining the matching rate and the inverse of the average brightness difference in the overlapping area between the divided images, and can stitch the divided images to the position where the matching score is maximized.

[0122] A vision sensor-based real-time vehicle shape analysis device (100) for detecting improper loading can calculate distance information between the vision sensor (110) and the vehicle using a triangulation method based on the focal length and / or baseline based on the parallax information of the stitched image.

[0123] A vision sensor-based real-time vehicle shape analysis device (100) for detecting improper loading can calculate the specifications of a vehicle based on distance information and determine the vehicle as an improperly loaded vehicle if the calculated specifications of the vehicle exceed a preset standard.

[0124] In addition, all operation steps of the vehicle shape analysis method described above can be performed identically by the processor (120) of the vision sensor-based real-time vehicle shape analysis device (100) for detecting loading defects.

[0126] FIG. 14 is a block diagram showing the specific configuration of a vision sensor-based real-time vehicle shape analysis device for detecting loading defects illustrated in FIG. 1 according to an embodiment of the present invention.

[0127] Referring to FIG. 14, a vision sensor-based real-time vehicle shape analysis device (100) for detecting improper loading includes a vision sensor (110), a processor (120), and a storage unit (130).

[0128] The processor (120) controls the overall operation of the vision sensor-based real-time vehicle shape analysis device (100) for detecting loading defects.

[0129] Specifically, the processor (120) includes RAM (121), ROM (122), main CPU (123), graphics processing unit (124), first to n interfaces (125-1 to 125-n), and a bus (126).

[0130] RAM (121), ROM (122), main CPU (123), graphics processing unit (124), first to n interfaces (125-1 to 125-n), etc. can be connected to each other via a bus (126).

[0131] The first to n interfaces (125-1 to 125-n) are connected to the various components described above. One of the interfaces may be a network interface connected to an external device through a network.

[0132] The main CPU (123) accesses the storage unit (130) and performs booting using the O / S stored in the storage unit (130). Then, it performs various operations using various programs, content, data, etc. stored in the storage unit (130).

[0133] In particular, the main CPU (123) can analyze the shape of a moving vehicle in real time using data obtained from a vision sensor and calculate the vehicle's specifications based on the analysis results.

[0134] A set of instructions for booting the system is stored in the ROM (122). When a turn-on command is input and power is supplied, the main CPU (123) copies the O / S stored in the storage unit (130) to the RAM (121) according to the instructions stored in the ROM (122), and executes the O / S to boot the system. When booting is complete, the main CPU (123) copies various application programs stored in the storage unit (130) to the RAM (121), and executes the application programs copied to the RAM (121) to perform various operations.

[0135] The graphics processing unit (124) generates a screen containing various objects such as icons, images, and text using a calculation unit (not shown) and a rendering unit (not shown). The calculation unit (not shown) calculates attribute values ​​such as coordinate values, shape, size, and color for each object to be displayed according to the layout of the screen based on a received control command. The rendering unit (not shown) generates a screen with various layouts containing objects based on the attribute values ​​calculated by the calculation unit (not shown).

[0136] In particular, the graphics processing unit (124) can implement objects generated by the main CPU (123) into a GUI (Graphic User Interface), icon, user interface screen, etc.

[0137] Meanwhile, the operation of the above-described processor (120) can be performed by a program stored in the storage unit (130).

[0138] The storage unit (130) stores various data, such as an O / S (Operating System) software module for operating a vision sensor-based real-time vehicle shape analysis device (100) for detecting loading defects, and various multimedia content.

[0139] In particular, the storage unit (130) may include a software module for analyzing the shape of a moving vehicle in real time using data acquired from a vision sensor and calculating the vehicle's specifications based on the analysis results.

[0141] FIG. 15 is a drawing relating to a software module stored in a storage unit according to an embodiment of the present invention.

[0142] Referring to FIG. 15, the storage unit (130) may store programs such as a data receiving module (131) and a vehicle shape analysis module (132).

[0143] Meanwhile, the operation of the processor (120) described above can be performed by a program stored in the storage unit (130). Below, the detailed operation of the processor (120) using the program stored in the storage unit (130) will be explained in detail.

[0144] The data receiving module (131) can receive data for calculating the specifications of the vehicle obtained from the vision sensor (110).

[0145] The vehicle shape analysis module (132) can analyze the shape of the vehicle based on data acquired from the vision sensor (110) and calculate the specifications of the vehicle based on the analysis results. At this time, the data acquired from the vision sensor (110) may include image data acquired to calculate the specifications of the vehicle, and the image data may include a depth map generated by extracting depth information based on disparity information, focal length, and baseline calculated from the left and right cameras.

[0146] The vehicle shape analysis module (132) can generate depth information by calculating the parallax from the data. At this time, the depth information may refer to the distance value from the vision sensor (110) to the vehicle, i.e., distance information.

[0147] The vehicle shape analysis module (132) can analyze the road surface brightness value of a frame where no vehicle is present to adjust the exposure time, and use this to update the background image from the frame.

[0148] The vehicle shape analysis module (132) can determine whether a vehicle has entered by analyzing the depth difference between the depth image of the current frame and the background image, and can generate a mask image of the vehicle with pixels greater than or equal to a preset height relative to the ground.

[0149] The vehicle shape analysis module (132) can sequentially collect frames with applied mask images as segmented images while the vehicle is present, and can stop collecting segmented images when the vehicle leaves.

[0150] The vehicle shape analysis module (132) can detect edges for the vehicle region of the segmented image and enhance the contour of the vehicle using at least one of bidirectional filtering and morphology operation. At this time, the morphology operation may be either a gradient and / or a top-hat.

[0151] The vehicle shape analysis module (132) can remove edges that are not valid for stitching by analyzing the brightness of edge pixels as a histogram. In this case, according to the embodiment, stitching is a process of reconstructing a part of the vehicle's shape observed in individual frames into a continuous and integrated single image by matching feature information of each image based on an overlapping area between multiple segmented images acquired while the vehicle is moving. In the present invention, this may refer to an algorithmic technology that determines the optimal matching position between frames using edge-based feature analysis, histogram-based edge filtering, and calculation of a matching score based on a matching rate and brightness difference, thereby accurately restoring the vehicle's overall contour and depth information.

[0152] The vehicle shape analysis module (132) can calculate a matching score by combining the matching rate and the inverse of the average brightness difference in the overlapping area between the segmented images, and stitch the segmented images to the position where the matching score is maximized.

[0153] The vehicle shape analysis module (132) can calculate distance information between the vision sensor (110) and the vehicle using a triangulation method based on the parallax information of the stitched image and / or the focal length and / or baseline.

[0154] The vehicle shape analysis module (132) calculates the vehicle's specifications based on distance information, and if the calculated vehicle's specifications exceed a preset standard, it can determine that the vehicle is a vehicle with poor loading.

[0155] Meanwhile, a non-transitory computer-readable medium may be provided that stores a program for sequentially performing the vehicle shape analysis method according to the present invention.

[0156] A non-transient readable medium refers to a medium that stores data semi-permanently and can be read by a device, rather than a medium that stores data for a short period of time, such as a register, cache, or memory. Specifically, the various applications or programs described above may be stored and provided on non-transient readable media such as CDs, DVDs, hard disks, Blu-ray discs, USBs, memory cards, and ROMs.

[0157] In addition, although a bus is not shown in the above-described block diagram illustrating a vision sensor-based real-time vehicle shape analysis device (100) for detecting improper loading, a processor such as a CPU or a microprocessor may be further included to analyze the shape of a moving vehicle in real time using data acquired from a vision sensor and to calculate vehicle specification information based on the analysis results.

[0158] Furthermore, although preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above. It is understood that various modifications can be made by those skilled in the art without departing from the essence of the invention as claimed in the claims, and such modifications should not be understood individually from the technical spirit or perspective of the present invention. Explanation of the symbols

[0160] 100: Vision sensor-based real-time vehicle shape analysis device for detecting improper loading 110: Vision sensor 120 : Processor 130 : Storage unit

Claims

Claim 1 A real-time vehicle shape analysis device based on a vision sensor for detecting loading defects, comprising: a vehicle shape analysis device for determining whether a vehicle traveling on a road has a loading defect; a vision sensor for acquiring data to calculate the specifications of the vehicle traveling; and a processor for analyzing the shape of the vehicle based on the data acquired from the vision sensor and calculating the specifications of the vehicle based on the analysis result; wherein the processor analyzes the depth difference between a depth image of a current frame and a background image to determine whether a vehicle traveling on the road has entered, generates a mask image for a vehicle area using pixels greater than or equal to a preset height relative to the ground, sequentially collects frames to which the mask image is applied as segmented images while the vehicle is present, terminates the collection of segmented images when the vehicle leaves the shooting area, detects edges of the vehicle area for each of the segmented images, removes edges among the detected edges that are not valid for stitching to enhance the contour of the vehicle, calculates a matching score by combining the matching rate and the inverse of the average brightness difference in the overlapping area between the segmented images, and reconstructs the shape of the vehicle by stitching the segmented images based on the position where the matching score is maximum. Claim 2 A vision sensor-based real-time vehicle shape analysis device for detecting loading defects, characterized in that, in claim 1, the processor calculates a time difference from the data to generate depth information. Claim 3 A vision sensor-based real-time vehicle shape analysis device for detecting improper loading, characterized in that, in claim 2, the processor analyzes the road surface brightness value of a frame where the vehicle is not present to adjust the exposure time, and uses this to update the background image from the frame. Claim 4 delete Claim 5 delete Claim 6 A vision sensor-based real-time vehicle shape analysis device for detecting improper loading, characterized in that, in claim 1, the processor detects edges for a vehicle region of the segmented image and enhances the contour of the vehicle using at least one of bidirectional filtering and morphological operation. Claim 7 A vision sensor-based real-time vehicle shape analysis device for detecting loading defects according to claim 6, wherein the processor analyzes the brightness of the edge pixels as a histogram to remove edges that are not valid for stitching. Claim 8 delete Claim 9 A vision sensor-based real-time vehicle shape analysis device for detecting improper loading, characterized in that, in claim 1, the processor calculates distance information between the vision sensor and the vehicle using a triangulation method based on focal length and / or baselines based on parallax information of the stitching image. Claim 10 A vision sensor-based real-time vehicle shape analysis device for detecting improper loading, characterized in that, in claim 9, the processor calculates the specifications of a vehicle based on the distance information and determines the vehicle as an improperly loaded vehicle when the calculated specifications of the vehicle exceed a preset standard. Claim 11 A method for analyzing vehicle shape in a vision sensor-based real-time vehicle shape analysis device for cracking down on improper loading, comprising: a step of receiving data for calculating the specifications of a vehicle driving on a road; and a step of analyzing the shape of the vehicle based on the data for calculating the specifications of the vehicle, and calculating the specifications of the vehicle based on the analysis result; wherein the step of calculating the specifications of the vehicle comprises: determining whether a driving vehicle has entered by analyzing the depth difference between a depth image of a current frame and a background image, generating a mask image for a vehicle area using pixels greater than or equal to a preset height relative to the ground, sequentially collecting frames to which the mask image is applied as segmented images while the vehicle is present, terminating the collection of segmented images when the vehicle moves out of the shooting area, detecting edges of the vehicle area for each of the segmented images, reinforcing the contour of the vehicle by removing edges among the detected edges that are not valid for stitching, calculating a matching score by combining the matching rate and the inverse of the average brightness difference in the overlapping area between the segmented images, and reconstructing the shape of the vehicle by stitching the segmented images based on the position where the matching score is maximum. Claim 12 A computer-readable recording medium storing a program for executing the vehicle shape analysis method described in claim 11 on a computer.

Images