Scattered object real-time detection method and system based on intelligent lamp pole and storage medium

Abstract

The invention discloses a scattered object real-time detection method and system based on an intelligent lamp post and a storage medium, and the method comprises the steps: carrying out the target segmentation of a generated disparity map, and generating a first segmentation target; calculating the size information of the first segmentation target based on the image coordinate of the target frame of the first segmentation target and the corresponding parallax value, and marking the first segmentation target of which the size information is smaller than a size threshold as a second segmentation target; acquiring position information of a second segmentation target in the continuous multi-frame disparity map according to a time sequence, and fitting a motion track of the second segmentation target based on the position information; and judging whether the second segmentation target is a lost object or not based on the motion trail. According to the technical scheme of the invention, the suspected second segmentation target is segmented from the disparity map, and the motion track of the second segmentation target is obtained through fitting based on the position information of the second segmentation target, so that whether the second segmentation target is a lost object or not is judged, and the real-time detection of the lost object on a road on the basis of not completely depending on a network model is facilitated.

Description

Technical Field

[0001] The present application relates to the technical field of smart cities, and more specifically, to a real-time detection method for spilled objects based on smart lamp poles, a real-time detection system for spilled objects based on smart lamp poles, and a computer storage medium. Background Art

[0002] In smart cities, it is necessary to monitor the safety of roads in real time to ensure that vehicles can travel safely and smoothly. Objects spilled during the driving of vehicles (especially transport vehicles) are an important factor affecting road safety due to their suddenness and diversity. Therefore, it is of great significance to accurately and real-time detect spilled objects from images, notify road monitoring personnel in time, take appropriate treatment measures, eliminate safety hazards as soon as possible, and ensure safe and smooth roads.

[0003] In the prior art, a network model is usually used for image recognition, and the spilled objects are regarded as a kind of obstacles. The fallen spilled objects are identified as obstacles through obstacle detection. For example, in the Chinese invention patent application "Road Spilled Object Detection Method, Device and Electronic Equipment" with publication number CN116381698A, after the roadside laser radar obtains the point cloud image, different detection models are used to detect road obstacles.

[0004] Such methods usually have the following problems:

[0005] 1. Non-real-time: Existing detection methods are usually based on obstacle detection, and when the spilled objects are detected, they have already fallen to the ground, and no warning can be given during their falling process;

[0006] 2. High cost: Existing detection methods are usually based on network models, and the training of network models requires a large amount of labeled data. The hardware operation cost of the model and the manual labeling cost of the data will affect the final implementation of the project;

[0007] 3. Unstable detection effect: The detection effect of the network model depends largely on the construction of the training set. However, due to the diversity and suddenness of the types of spilled objects, the trained network model is often unable to directly and accurately identify unknown spilled objects on the road, resulting in low accuracy and reliability of spilled object detection. Summary of the invention

[0008] The purpose of this application is to detect scattered objects on the road in real time without relying entirely on a network model.

[0009] The technical solution of the first aspect of the present application is: a real-time detection method for spilled objects based on smart lamp poles is provided, the method comprising: step 1: performing target segmentation on the generated disparity map to generate a first segmented target; step 2: calculating the size information of the first segmented target based on the image coordinates of the target frame of the first segmented target and the corresponding disparity value, and recording the first segmented target whose size information is less than the size threshold as the second segmented target; step 3: acquiring the position information of the second segmented target in the disparity map of multiple consecutive frames in time sequence, and fitting the motion trajectory of the second segmented target based on the position information; step 4: judging whether the second segmented target is a spilled object based on the motion trajectory.

[0010] In any of the above technical solutions, further, the disparity map is generated by a binocular camera, and the binocular camera is installed on a smart lamp pole. Multiple smart lamp poles are arranged in sequence on both sides of the road. In step 1, it specifically includes: Step 101: Labeling the disparity maps generated by multiple binocular cameras at the same time, wherein the labels are determined by the numbers of the smart lamp poles on which the binocular cameras are installed, and the numbers increase sequentially along the direction of vehicle travel; Step 102: Performing target segmentation on the labeled disparity map, and calculating the spatial position of each segmented third segmentation target in the world coordinate system; Step 103: When it is determined that the difference between the spatial positions of the third segmentation targets in two frames of disparity maps with adjacent labels is less than or equal to the screening threshold, filtering out the third segmentation targets in the disparity map with a larger label, and recording the remaining third segmentation targets as the first segmentation targets.

[0011] In any of the above technical solutions, further, before step 1, the method also includes: step 111: converting the acquired sampled image into a grayscale image, and filtering the grayscale image to generate a filtered image; step 112: calculating an adaptive threshold for each pixel in the filtered image, wherein the adaptive threshold is determined by the grayscale value of the pixel and its corresponding weight, and the weight is determined by the absolute value of the difference between the grayscale value of the pixel and the grayscale mean in the corresponding pixel window; step 113: based on the adaptive threshold, using the Sobel edge detection algorithm to generate an initial edge image, and based on the initial edge image and the sampled image, using the scale-invariant feature transformation algorithm to generate a disparity map.

[0012] In any of the above technical solutions, the grayscale image is further filtered to generate a filtered image, which specifically includes: based on a preset filtering window, calculating the minimum grayscale value, maximum grayscale value and grayscale median corresponding to the current center point; when it is determined that the difference between the minimum grayscale value and the maximum grayscale value is less than the grayscale threshold, the grayscale value of the current center point is recorded as the center point grayscale value; when it is determined that the difference between the minimum grayscale value and the maximum grayscale value is greater than or equal to the grayscale threshold, judging whether the grayscale median is greater than the minimum grayscale value and less than the maximum grayscale value, if so, recording the grayscale median as the center point grayscale value, if not, increasing the size of the preset filtering window according to the preset step size, and recalculating the minimum grayscale value, maximum grayscale value and grayscale median corresponding to the current center point; based on the center point grayscale value, generating a filtered image.

[0013] In any of the above technical solutions, further, the disparity map is generated by a binocular camera, the binocular camera is installed on a smart light pole, and a plurality of smart light poles are arranged in sequence on both sides of the road. In step 113, specifically, it also includes: calibrating the binocular cameras installed on two adjacent smart light poles to determine the overlapping area of ​​the field of view of the two adjacent binocular cameras; based on the overlapping area of ​​the field of view, determining the area to be fused in the initial edge images corresponding to the front and rear binocular cameras, respectively, wherein the area to be fused corresponding to the front binocular camera is recorded as the first area to be fused, and the area to be fused corresponding to the rear binocular camera is recorded as the second area to be fused; calculating A difference map between the first area to be fused and the second area to be fused; based on the coordinates of any pixel point with a value of 1 in the difference map, the corresponding pixel point in the first area to be fused is marked and recorded as a pending edge point; according to the region growing method, within a preset interval, the number of edge points connected to the pending edge point is counted, and when the number is greater than or equal to a preset threshold, the pending edge point is recorded as a fused edge point, and the first area to be fused in the initial edge image is updated according to the fused edge point to generate a fused edge image; based on the fused edge image and the sampled image, a scale-invariant feature transformation algorithm is used to generate a disparity map.

[0014] In any of the above technical solutions, further, step 4 specifically includes: fitting the ground plane based on the disparity map; when the motion trajectory of the second segmented target and the fitted ground plane have an intersection, or, based on a preset time interval, when the distance between the second segmented target and the ground plane decreases, and from a certain moment, the distance is less than a preset distance threshold, the second segmented target is determined to be a spilled object.

[0015] In any of the above technical solutions, the method further includes: when it is determined that the size information of any first segmented target belongs to the vehicle size threshold interval, the first segmented target is recorded as a vehicle target; the position information of the vehicle target is calculated, and when it is determined that the position information of the vehicle target is located behind the position information of the second segmented target, it is determined whether the vehicle target has an avoidance behavior, if so, the second segmented target is marked as a spilled object, if not, the second segmented target is marked as a non-spilled object.

[0016] The technical solution of the second aspect of the present application is: a real-time detection system for spilled objects based on a smart lamp pole is provided, the system comprising: a first segmentation unit, the first segmentation unit is configured to perform target segmentation on the generated disparity map to generate a first segmented target; a second segmentation unit, the second segmentation unit is configured to calculate the size information of the first segmented target based on the image coordinates of the target frame of the first segmented target and the corresponding disparity value, and record the first segmented target whose size information is less than the size threshold as the second segmented target; a trajectory fitting unit, the trajectory fitting unit is configured to obtain the position information of the second segmented target in the disparity map of multiple consecutive frames in time sequence, and fit the motion trajectory of the second segmented target based on the position information; a spillage determination unit, the spillage determination unit is configured to determine whether the second segmented target is a spilled object based on the motion trajectory.

[0017] In any of the above technical solutions, further, the spill judgment unit is configured to: fit the ground plane based on the disparity map; when the motion trajectory of the second segmented target and the fitted ground plane have an intersection, or, according to a preset time interval, when the distance between the second segmented target and the ground plane decreases, judge the second segmented target as a spilled object.

[0018] The technical solution of the third aspect of the present application is: a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, a real-time detection method for spilled objects based on a smart lamp pole is implemented as any one of the technical solutions of the first aspect of the present application.

[0019] The beneficial effects of this application are:

[0020] The technical solution in the present application can be used to segment and screen out the second segmented target of the suspected spilled object in the disparity map, and then fit its motion trajectory based on its position information and the time information of the disparity frame to determine whether the second segmented target is a spilled object. There is no need to use a network model to identify the target in the image, and the spilled objects on the road can be detected without relying entirely on the network model, which helps to reduce the overall cost of the project, and the detection can be completed during the falling process, with high real-time performance. In addition, the above technical solution only detects spilled objects based on the position and motion trajectory of the target, without identifying the category of the object, and can well cope with the detection of multiple types of temporary and sudden spilled objects, which helps to improve the accuracy and reliability of spilled object detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The advantages of the above and / or additional aspects of the present application will become apparent and easily understood in the description of the embodiments in conjunction with the following drawings, in which:

[0022] Figure 1 is a schematic flow chart of a real-time detection method for spilled objects based on a smart lamp pole according to an embodiment of the present application;

[0023] Figure 2 is a schematic diagram of a depth z calculation principle according to an embodiment of the present application;

[0024] Figure 3 is a schematic diagram of the motion trajectory of a spilled object according to an embodiment of the present application;

[0025] Figure 4 is a schematic diagram of a field of view overlap region according to an embodiment of the present application;

[0026] Figure 5 is a schematic diagram of fusion edge points according to an embodiment of the present application;

[0027] Figure 6 It is a schematic block diagram of a real-time detection system for spilled objects based on a smart lamp pole according to an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0029] In the following description, many specific details are elaborated to facilitate a full understanding of the present application. However, the present application may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present application is not limited to the specific embodiments disclosed below.

[0030] In the process of building a smart city, a smart lamp pole equipped with a binocular camera can be used as a sensing device. After the binocular camera is installed and calibrated, it can output a sampled image within the field of view and a corresponding disparity map. The sampled image includes a left image obtained by the left camera and a right image obtained by the right camera, and the disparity map is calculated from the left image and the right image based on the disparity calculation principle.

[0031] It should be noted that when calculating the disparity map, the left image is usually selected as the basis. Therefore, the binocular camera can use the left image and the disparity map as output. The left image can be used for target recognition, and the depth information of each target can be obtained from the disparity map. Then, the spatial information of each target can be obtained through coordinate transformation.

[0032] Considering that when detecting spilled objects on the road, the most important thing is to obtain information such as the location and size of the spilled objects in order to alert passing vehicles; the type of spilled objects (such as cartons, plastic bags) can usually be used as secondary information, mainly to facilitate road monitoring personnel to remove them.

[0033] Therefore, scattered objects on the road can be detected in real time based on the disparity map without relying on the network model, and information such as the location and size of the scattered objects can be obtained.

[0034] Embodiment 1:

[0035] like Figure 1 As shown, this embodiment provides a real-time detection method for spilled objects based on a smart lamp pole, the method comprising:

[0036] Step 1: Perform target segmentation on the generated disparity map to generate the first segmentation target;

[0037] Specifically, after the binocular camera generates a disparity map, the disparity map can be segmented using a segmentation algorithm such as clustering and connected domain analysis, thereby segmenting multiple targets, such as vehicles, pedestrians, trees, etc., and these segmented targets are used as the first segmentation targets.

[0038] Step 2: Based on the image coordinates of the target frame of the first segmentation target and the corresponding disparity value, the size information of the first segmentation target is calculated, and the first segmentation target whose size information is smaller than the size threshold is recorded as the second segmentation target;

[0039] Specifically, Figure 2 As shown in the figure, for the disparity value of a certain disparity point, combined with the internal parameters of the binocular camera, such as the baseline distance B, focal length f, optical center coordinates, etc., the depth z between the disparity point and the plane where the binocular camera is located (xoy plane) can be calculated. The corresponding calculation formula is:

[0040]

[0041] Where disp is the disparity value.

[0042] After obtaining the depth z, based on the pixel coordinates (u, v) of the disparity point in the left image and the corresponding disparity value, the coordinates (x w ,y w ,z w ), i.e., location information, the corresponding calculation formula is:

[0043]

[0044] In the formula, (u 0 ,v 0 ) is the optical center coordinate of the binocular camera.

[0045] Therefore, the above method can be used to calculate the size information of the first segmentation target, such as width and length, based on the image coordinates of the target frame of the first segmentation target and the corresponding disparity value, where the width corresponds to the x-axis direction and the length corresponds to the y-axis direction. The specific value can be obtained by the distance between the two coordinate points.

[0046] Different targets, such as vehicles, cartons, and pedestrians, have different sizes. Therefore, a size threshold is set for common spilled objects. For any first segmented target, if its size information is greater than or equal to this size threshold, it means that its size is large and it is not a spilled object; if its size information is less than this size threshold, it means that it is a potential spilled object. Therefore, this type of first segmented target is recorded as a second segmented target and needs to be judged later.

[0047] It should be noted that the size threshold may be one of the length and the width, or a combination of different ones.

[0048] Step 3: Acquire the position information of the second segmentation target in the disparity map of the continuous multiple frames in time sequence, and fit the motion trajectory of the second segmentation target based on the position information;

[0049] Step 4: Based on the motion trajectory, determine whether the second segmented target is a spilled object.

[0050] Specifically, after obtaining the second segmentation target (potential spilled object), target tracking can be performed in the multi-frame disparity map according to the size information of the second segmentation target, so as to obtain the disparity point coordinates of the second segmentation target in the continuous multi-frame disparity map according to the time sequence. The disparity point coordinates can be the coordinates of the upper left corner of its target frame or the center position of the target frame. Then, through the above calculation formula, the position information (xw ,y w ,z w ).

[0051] After that, the motion trajectory of the second segmentation target can be obtained by fitting based on the time corresponding to each frame and the position of the second segmentation target. In the fitting process, considering that the falling process of the spilled object will be affected by the initial velocity, gravity, air resistance and other factors, a polynomial fitting method can be used, which is set as:

[0052] y=a n x n +a n-1 x n-1 +...+a 1 x+a 0

[0053] Where y is the position information of the second segmentation target, x is the time corresponding to each frame of the disparity map, and a n to a 0 is the fitting coefficient. In the process of solving the fitting coefficient, the least squares method can be used for iterative optimization. The coefficient vector of the fitting coefficient is continuously adjusted by minimizing the loss function Loss to find the fitting coefficient of the best fitting polynomial. The calculation formula of the loss function Loss and the least squares method is as follows:

[0054]

[0055] In the formula, y i As the actual value of the i-th observation point, is the predicted value of the ith observation point, and N is the total number of observation points.

[0056] The calculation formula for the fitting coefficients is obtained by solving the normal equations using the least squares method:

[0057] (X T X)a=X T y

[0058]

[0059] a=[a 0 , a 1 , ..., a n ] T

[0060] Where a is the coefficient vector, y is the observation vector, and X is a matrix constructed according to the highest order n of the fitted polynomial, with each row corresponding to an observation point x i , i ranges from 1 to N, and each column corresponds to x i The power of .

[0061] The polynomial coefficient vector a obtained through the above steps will minimize the loss function Loss. Substituting the obtained coefficient vector a into the polynomial model can obtain the best fitting motion trajectory, such as Figure 3 As shown, this trajectory is as close as possible to all observed data points.

[0062] Finally, based on the above-mentioned motion trajectory and the set rules, it can be determined whether the second segmentation target is a spilled object.

[0063] For example, when the fitted motion trajectory is an irregular upward curve, the second segmentation target may be a plastic bag, which has a weak impact on road traffic safety and may not be determined as a spilled object;

[0064] When the fitted motion trajectory is an irregular downward curve and does not move after contacting the ground, the second segmentation target may be some flexible scattered objects, such as clothing, school bags, luggage bags, etc.

[0065] When the fitted motion trajectory is an irregular downward curve and there are still several small curves after contacting the ground, the second segmentation target may be some rigid objects (which may bounce or slide after contacting the ground), such as cartons, luggage rolls, etc.

[0066] It should be noted that when a spilled object is detected, since the disparity map corresponds to the left image one by one, the corresponding left image can be displayed to the corresponding road monitoring personnel so that they can organize personnel to clear the obstacle.

[0067] Through the technical solution in this embodiment, the second segmentation target of the suspected spilled object is segmented and screened out in the disparity map, and then based on its position information and the time information of the disparity frame, its motion trajectory can be fitted to determine whether the second segmentation target is a spilled object. There is no need to use a network model to identify the target in the image, and the spilled objects on the road can be detected without relying entirely on the network model, which helps to reduce the overall cost of the project, and the detection can be completed during the falling process, with high real-time performance. In addition, the above technical solution only detects spilled objects based on the position and motion trajectory of the target, without identifying the category of the object, and can well cope with the detection of multiple types of temporary and sudden spilled objects, which helps to improve the accuracy and reliability of spilled object detection.

[0068] In any of the above embodiments, further, the disparity map is generated by a binocular camera, the binocular camera is installed on a smart light pole, and a plurality of smart light poles are sequentially arranged on both sides of the road. Step 1 specifically includes:

[0069] Step 101: labeling the disparity maps generated by multiple binocular cameras at the same time, wherein the labels are determined by the numbers of the smart lamp poles on which the binocular cameras are installed, and the numbers increase in sequence along the direction of vehicle travel;

[0070] Step 102: performing object segmentation on the labeled disparity map, and calculating the spatial position of each segmented third segmentation object in the world coordinate system;

[0071] Step 103: When it is determined that the difference between the spatial positions of the third segmentation targets in the disparity images of two frames with adjacent labels is less than or equal to the screening threshold, the third segmentation targets in the disparity images with larger labels are filtered out, and the remaining third segmentation targets are recorded as first segmentation targets.

[0072] Specifically, considering that there are usually multiple smart lamp poles on both sides of the road, and the fields of view of the binocular cameras in each smart lamp pole may overlap, the same target may appear in the fields of view of two adjacent binocular cameras. Therefore, the multi-frame disparity maps generated by multiple binocular cameras acquired at the same time can be numbered, where the number is determined by the number of the smart lamp pole on which the binocular camera is installed, and the number can be set to increase sequentially along the direction of vehicle travel.

[0073] like Figure 4 As shown, in simple terms, the disparity maps collected by each binocular camera are numbered in sequence according to the number of the smart light pole. For example, the disparity map obtained by the i-th smart light pole (binocular camera) at time t is numbered i t , and the parallax icon number obtained by the i+1th smart light pole is (i+1) t ; Correspondingly, at time t+1, the labels of the two disparity images are i and t+1 、(i+1) t+1 .

[0074] The binocular cameras of two adjacent smart lamp poles will have an overlapping field of view. When there is a target in this area, it will be presented in two disparity maps respectively, such as "Target 2".

[0075] At this time, if the spilled object detection is performed directly, it will be judged as "twice", so one of the "targets 2" needs to be filtered out.

[0076] For multiple disparity images acquired at the same time, the image coordinates of the upper left corner of the target frame of the second segmentation target in the two disparity images with adjacent labels are selected to calculate the position information, and then the position information in the camera coordinate system is transformed into the spatial position in the world coordinate system by using the coordinate system transformation method, where the origin coordinates of the world coordinate system can be set according to actual needs.

[0077] Therefore, when the difference between the two corresponding spatial positions of target 2 and target 2' (such as the distance difference) is less than or equal to the screening threshold, it can be considered that target 2 and target 2' should be the same target.

[0078] Since in the parallax calculation process, “the accuracy of near parallax is higher than that of far parallax,” when the field of view of the binocular camera is tilted forward, the i t The disparity value of "target 2" in the frame disparity map is greater than that of the (i+1)th t The accuracy of the frame disparity map, therefore, filter out the (i+1)th t The third segmentation target (target 2') and the corresponding position information in the frame disparity map are retained. t The third segmentation target (target 2) and the corresponding position information in the frame disparity map are recorded as the first segmentation target.

[0079] In any of the above embodiments, further, before step 1, the method further includes:

[0080] Step 111: converting the acquired sampled image into a grayscale image, and filtering the grayscale image to generate a filtered image;

[0081] Step 112: Calculate an adaptive threshold value for each pixel in the filtered image, wherein the adaptive threshold value is determined by the gray value of the pixel and its corresponding weight, and the weight is determined by the absolute value of the difference between the gray value of the pixel and the gray mean value in the corresponding pixel window;

[0082] Step 113: Based on the adaptive threshold, a Sobel edge detection algorithm is used to generate an initial edge image, and based on the initial edge image and the sampled image, a scale-invariant feature transformation algorithm is used to generate a disparity map.

[0083] Specifically, in the process of calculating the disparity, a scale-invariant feature transformation algorithm can be used to extract feature points from the left image and the right image respectively, and then the feature points extracted from the left and right images are matched to find the corresponding feature point pairs, and then their horizontal offsets, i.e., the disparity values, are calculated to finally obtain the disparity map.

[0084] The feature points extracted by this type of method are usually edge points and corner points in the image. Therefore, we can first use an edge detection algorithm based on the four-directional Sobel edge detection operator to extract the edge points in the left and right images and generate an initial edge image. Then, the initial edge image and the sampled image are used as input together to ensure the accuracy of feature point extraction by the scale-invariant feature transformation algorithm.

[0085] In the process of generating the initial edge image, the edge image generation from the left image is taken as an example. First, the left image (sampled image) is converted into a grayscale image and then filtered. The filtering method can be Gaussian filtering, adaptive median filtering, and Gaussian smoothing can also be performed on the converted grayscale image.

[0086] Taking into account the large difference in grayscale values ​​between the acquired images due to environmental influences, the color intensity of different areas of the same image is also different. The grayscale value of the overall pixel in the darker area is quite different from that in the brighter area, and the gradient value also has a difference in amplitude. Therefore, it is necessary to adaptively adjust the threshold according to the specific situation of the regional pixels.

[0087] Therefore, this embodiment proposes to calculate the weight of each pixel point participating in the mean calculation based on the distance between it and the mean. The larger the distance, the smaller its proportion, which can reflect the overall brightness level in the local area, has good adaptability to brightness changes, and suppresses interference from overly bright or dark areas. At the same time, it can effectively avoid the influence of individual points with large deviations in the window, so that the data can better reflect the level of most pixel points in the window and reduce the probability of false detection.

[0088] In the process of calculating the adaptive threshold of each pixel in the filtered image, firstly, in a 3×3 pixel window centered at any pixel i, the mean grayscale value of the 9 pixels in the window is calculated. Then calculate the mean The absolute value of the difference between the grayscale values ​​of these 9 pixels Among them, m is the label of these 9 pixels. If the 9 pixels in the 3×3 pixel window are sorted from left to right and from top to bottom, then when m=5, it corresponds to the pixel i.

[0089] According to the absolute value of the difference Calculate the weight w of pixel i i , the larger the difference, the smaller the weight, and the corresponding calculation formula is:

[0090]

[0091] Finally, with z i w i as an adaptive threshold.

[0092] Then, based on the Sobel edge detection algorithm, the four grayscale value change gradients G of a pixel can be obtained. 0 , G 45 , G 90 , G 135 , and then use the following formula to weight the sum of these four gray value change gradients to calculate the composite gradient amplitude G(x,y) and gradient direction θ(x,y) of the pixel:

[0093]

[0094] Finally, the Otsu (maximum inter-class variance method) algorithm is used to determine the edge points. In the determination process, each pixel is checked along the gradient direction. If the gradient amplitude of this pixel is not the local maximum in this direction (compared with the adjacent pixels in this direction), the gradient amplitude of this pixel will be suppressed, that is, set to zero or ignored.

[0095] Then scan each pixel in the filtered image, compare its gradient amplitude G(x, y) with the adaptive threshold, and record the pixel points greater than the adaptive threshold as strong edges, which are displayed as 1 in the binary image; the pixel points less than the adaptive threshold but greater than 1 / 2 of the adaptive threshold are recorded as weak edges. If the weak edge is connected to the strong edge in its 3×3 neighborhood, it is retained as 1, otherwise it is discarded and recorded as 0; the pixel points less than 1 / 2 of the adaptive threshold are directly recorded as 0, and the obtained binary image is the edge image.

[0096] In any of the above embodiments, further, in order to effectively remove linear and nonlinear noise of the image, a hybrid filter combining Gaussian filtering and adaptive median filtering is used to filter the grayscale image to generate a filtered image, specifically including:

[0097] Based on the preset filtering window, calculate the minimum grayscale value, maximum grayscale value and grayscale median corresponding to the current center point;

[0098] Specifically, the size of the preset filter window is first set to a small size, such as 5×5, 10×10, and then the maximum size of the preset filter window is set, such as 20×20, 30×30. Then, the sliding window method is used to calculate the minimum grayscale value, maximum grayscale value and grayscale median corresponding to the current center point in each sliding window when the sliding window slides, which are recorded as f min 、f max 、f med .

[0099] When the difference between the minimum grayscale value and the maximum grayscale value is determined to be less than the grayscale threshold, the grayscale value of the current center point is recorded as the center point grayscale value;

[0100] When it is determined that the difference between the minimum grayscale value and the maximum grayscale value is greater than or equal to the grayscale threshold, determine whether the grayscale median is greater than the minimum grayscale value and less than the maximum grayscale value. If so, record the grayscale median as the grayscale value of the center point. If not, increase the size of the preset filter window according to the preset step size, and recalculate the minimum grayscale value, maximum grayscale value and grayscale median corresponding to the current center point;

[0101] Specifically, a grayscale threshold T is set, and based on the minimum grayscale value, the maximum grayscale value, and the grayscale median value corresponding to the current center point within the window, the center point grayscale value corresponding to the window is determined. This process can be as follows:

[0102] If f max -f min <T, indicating that the grayscale values of the pixel points within the window are evenly distributed and there are no noise points, then the grayscale value f of the current center point xy is used as the center point grayscale value of the window;

[0103] If f max -f min ≥T and f min <f med <f max , it indicates that there are noise points within the window, and the grayscale median value f med is not a noise point, then the grayscale median value f med is recorded as the center point grayscale value of this window;

[0104] If f max -f min ≥T, but does not satisfy f min <f med <f max , then the size of the preset filtering window is increased according to the preset step size, and the above process is repeated.

[0105] It should be noted that in order to avoid the preset filtering window from being too large, this embodiment also sets the maximum size of the preset filtering window. If the center point grayscale value still cannot be obtained when the preset filtering window is increased to the maximum size, then the average grayscale value of all points within the preset filtering window of the current maximum size is used as the center point grayscale value of this window.

[0106] Based on the center point grayscale value determined by the above process, a filtered image is generated.

[0107] In any of the above embodiments, further, the disparity map is generated by a binocular camera. The binocular camera is installed on a smart light pole, and multiple smart light poles are sequentially arranged on both sides of the road. In step 113, it specifically further includes:

[0108] Calibrate the binocular cameras installed on two adjacent smart light poles to determine the overlapping field of view area of the two adjacent binocular cameras;

[0109] Based on the overlapping field of view area, determine the regions to be fused in the initial edge images corresponding to the front and rear binocular cameras respectively. Among them, the region to be fused corresponding to the previous binocular camera is denoted as the first region to be fused, and the region to be fused corresponding to the latter binocular camera is denoted as the second region to be fused;

[0110] Calculating a difference map between the first region to be fused and the second region to be fused;

[0111] Specifically, the areas to be integrated corresponding to the i-th and i+1-th smart lamp poles are recorded as the first areas to be integrated E. i , the second area to be fused E i+1 , perform a difference operation on the initial edge image areas corresponding to the two areas. If the position of a certain pixel in the two areas is both "1" or "0", it indicates that the judgment of whether the pixel is an edge in the two areas is consistent, and the position of the corresponding pixel in the difference map is recorded as "0"; if it is inconsistent, the position of the corresponding pixel is recorded as "1". Through the above process, the difference map E corresponding to the pixel and the first area to be fused can be obtained. j .

[0112] Based on the coordinates of any pixel with a value of 1 in the difference map, the corresponding pixel in the first area to be fused is marked as an undetermined edge point, such as Figure 5 The pixels marked with “*”, where the value is 1, are the edge points in the first area to be fused;

[0113] According to the region growing method, within a preset interval (such as 5×5), the number of edge points connected to the pending edge point is counted. When the number is greater than or equal to a preset threshold (such as 5), the pending edge point is recorded as a fused edge point. If the number of connected edge points is less than 5, the pending edge point is removed as a noise point. After that, the first region to be fused in the initial edge image is updated according to the marked fused edge points to generate a fused edge image.

[0114] Based on the fusion of edge image and sampling image, the disparity map is generated by using scale-invariant feature transformation algorithm.

[0115] In any of the above embodiments, further, in order to more accurately determine the spilled objects, the above step 4 specifically includes:

[0116] Fit the ground plane based on the disparity map;

[0117] When the motion trajectory of the second segmented target intersects with the fitted ground plane, or, based on a preset time interval, when the distance between the second segmented target and the ground plane decreases, and from a certain moment on, the distance is less than a preset distance threshold, the second segmented target is determined to be a spilled object.

[0118] Specifically, by analyzing the ground area in the disparity map, a preset plane fitting algorithm is used to establish a ground model; the plane fitting algorithm can use a conventional algorithm such as RANSAC (RANdom SAmple Consensus) to process the data of the ground area in the disparity map, and establish the ground plane equation Ax+By+Cz+D=0. RANSAC will randomly select points in the ground area and substitute them into the plane equations x, y, and z. When the number of random sampling is large enough, the coefficients A, B, C, and D in the plane equation can be parsed to obtain the equation representing the ground plane.

[0119] Considering that the spilled object will eventually fall to the ground, that is, there is an intersection between its motion trajectory and the ground plane, and some rigid objects may bounce after falling to the ground, so its trajectory may have multiple intersections with the ground. Therefore, the motion trajectory equation of the target can be substituted into the ground plane equation to find out whether there is an intersection. If there is at least one physically reasonable intersection, then it can be considered that the target is in contact with the ground, and the second segmentation target is determined to be a spilled object.

[0120] However, some objects (such as tarpaulins on trucks) do not fall directly when they fall. Therefore, the distance between them and the ground is used as a reference for judging whether they are spilled objects. That is, when the distance between the two decreases and, starting from a certain moment, the distance is less than a preset distance threshold, it is considered that the target is in the process of falling and is about to come into contact with the ground. The corresponding second segmented target is determined to be a spilled object. The three-dimensional spatial position (x w ,y w ,z w ) to the ground plane equation to calculate the distance from a point to a plane.

[0121] In any of the above embodiments, further, in the process of determining whether an object is spilled, the avoidance behavior of the following vehicle may be introduced, that is, if there is an avoidance behavior, the object in front is considered to be a spilled object. Therefore, the method further includes:

[0122] When it is determined that the size information of any first segmented object belongs to the vehicle size threshold interval, the first segmented object is recorded as a vehicle object;

[0123] Calculate the position information of the vehicle target. When it is determined that the position information of the vehicle target is behind the position information of the second segmented target, determine whether the vehicle target has an avoidance behavior. If so, mark the second segmented target as a spilled object. If not, mark the second segmented target as a non-spilled object.

[0124] Specifically, when determining that there is a vehicle target, if the corresponding vehicle is located behind the spilled object, the driver will respond with evasive behavior, such as changing lanes, slowing down, etc. If the vehicle is located in front, the driver can just drive normally. Therefore, in the process of determining the spilled object, the evasive behavior of the vehicle (the following vehicle) can be introduced to assist in the judgment.

[0125] This embodiment can continuously track the vehicle target based on its size and position information, fit its motion trajectory, and analyze its motion pattern (such as speed, acceleration, direction, etc.), and determine whether the vehicle target behind has slowed down, changed lanes (such as the vehicle's heading angle has changed significantly relative to the lane direction) and other behaviors when approaching the second segmented target. It can also identify whether there is an avoidance behavior, and the specific process will not be repeated here.

[0126] It should be noted that a threshold value of the number of vehicles that have avoided behavior may also be set, such as 3 vehicles, to avoid false detections caused by erroneous driving behavior of individual vehicles.

[0127] Embodiment 2:

[0128] like Figure 6 As shown, this embodiment provides a real-time detection system 100 for spilled objects based on a smart lamp pole, and the system 100 includes:

[0129] A first segmentation unit 10, the first segmentation unit 10 is configured to perform target segmentation on the generated disparity map to generate a first segmentation target;

[0130] The second segmentation unit 20 is configured to calculate the size information of the first segmentation target based on the image coordinates of the target frame of the first segmentation target and the corresponding disparity value, and record the first segmentation target whose size information is smaller than the size threshold as the second segmentation target;

[0131] A trajectory fitting unit 30, the trajectory fitting unit 30 is configured to obtain position information of the second segmentation target in the disparity map of the continuous multiple frames in a time sequence, and fit the motion trajectory of the second segmentation target based on the position information;

[0132] The spillage determination unit 40 is configured to determine whether the second segmented target is a spilled object based on the motion trajectory.

[0133] In any of the above embodiments, further, the disparity map is generated by a binocular camera, the binocular camera is installed on a smart light pole, a plurality of smart light poles are arranged in sequence on both sides of the road, and the plurality of smart light poles are connected to a first segmentation unit 10. The binocular camera in the smart light pole is only responsible for collecting left and right images and generating a disparity map. The first segmentation unit 10 performs the first segmentation target division. Therefore, the first segmentation unit 10 is further configured as follows:

[0134] The disparity maps generated by multiple binocular cameras at the same time are numbered, where the numbers are determined by the numbers of the smart lamp poles on which the binocular cameras are installed, and the numbers increase in sequence along the direction of vehicle travel;

[0135] Performing target segmentation on the labeled disparity map, and calculating the spatial position of each segmented third segmentation target in the world coordinate system;

[0136] When the difference between the spatial positions of the third segmentation targets in the disparity images of two frames with adjacent labels is determined to be less than or equal to the screening threshold, the third segmentation targets in the disparity images with larger labels are filtered out, and the remaining third segmentation targets are recorded as the first segmentation targets.

[0137] In any of the above embodiments, further, a data operation unit is provided in the binocular camera for processing the left and right images and calculating the corresponding disparity map, and the binocular camera is configured as follows:

[0138] Convert the acquired sampled image into a grayscale image, and filter the grayscale image to generate a filtered image;

[0139] Calculate the adaptive threshold of each pixel in the filtered image, wherein the adaptive threshold is determined by the gray value of the pixel and its corresponding weight, and the weight is determined by the absolute value of the difference between the gray value of the pixel and the gray mean in the corresponding pixel window;

[0140] Based on the adaptive threshold, the Sobel edge detection algorithm is used to generate an initial edge image, and based on the initial edge image and the sampled image, a scale-invariant feature transformation algorithm is used to generate a disparity map.

[0141] In any of the above embodiments, further, in order to effectively remove linear and nonlinear noise of the image, a hybrid filter combining Gaussian filtering and adaptive median filtering is used, and the above binocular camera is also configured as follows:

[0142] Based on the preset filtering window, calculate the minimum grayscale value, maximum grayscale value and grayscale median corresponding to the current center point;

[0143] Specifically, the size of the preset filter window is first set to a small size, such as 5×5, 10×10, and then the maximum size of the preset filter window is set, such as 20×20, 30×30. Then, the sliding window method is used to calculate the minimum grayscale value, maximum grayscale value and grayscale median corresponding to the current center point in each sliding window when the sliding window slides, which are recorded as f min 、f max 、f med .

[0144] When it is determined that the difference between the minimum gray value and the maximum gray value is less than the gray threshold, the gray value of the current center point is recorded as the center point gray value;

[0145] When it is determined that the difference between the minimum gray value and the maximum gray value is greater than or equal to the gray threshold, it is judged whether the median gray value is greater than the minimum gray value and less than the maximum gray value. If so, the median gray value is recorded as the center point gray value. If not, the size of the preset filtering window is increased according to the preset step length, and the minimum gray value, the maximum gray value and the median gray value corresponding to the current center point are recalculated;

[0146] Specifically, a gray threshold T is set, and based on the minimum gray value, the maximum gray value and the median gray value corresponding to the current center point within the window, the center point gray value corresponding to the window is determined. This process can be as follows:

[0147] If f max -f min <T, indicating that the gray values of the pixels within the window are evenly distributed and there are no noise points, then the gray value f xy of the current center point is used as the center point gray value of the window;

[0148] If f max -f min ≥T and f min <f med <f max , it indicates that there are noise points within the window, and the median gray value f med is not a noise point, then the median gray value f med is recorded as the center point gray value of the window;

[0149] If f max -f min ≥T, but does not satisfy f min <f med <f max , then the size of the preset filtering window is increased according to the preset step length, and the above process is repeated.

[0150] It should be noted that in order to avoid the preset filtering window from being too large, the maximum size of the preset filtering window is also set in this embodiment. If the center point gray value still cannot be obtained when the preset filtering window is increased to the maximum size, the average gray value of all points within the preset filtering window of the current maximum size is used as the center point gray value of the window.

[0151] Based on the center point gray value determined by the above process, a filtered image is generated.

[0152] In any of the above embodiments, further, the disparity map is generated by a binocular camera. The binocular camera is installed on a smart light pole, and multiple smart light poles are sequentially arranged on both sides of the road. The above binocular camera is further configured as:

[0153] Calibrate the binocular cameras installed on two adjacent smart light poles to determine the overlapping area of ​​the field of view of the two adjacent binocular cameras;

[0154] Based on the overlapping area of ​​the field of view, the area to be fused is determined in the initial edge images corresponding to the front and rear binocular cameras, respectively, wherein the area to be fused corresponding to the front binocular camera is recorded as the first area to be fused, and the area to be fused corresponding to the rear binocular camera is recorded as the second area to be fused;

[0155] Calculating a difference map between the first region to be fused and the second region to be fused;

[0156] Specifically, the areas to be integrated corresponding to the i-th and i+1-th smart lamp poles are recorded as the first areas to be integrated E. i , the second area to be fused E i+1 , perform a difference operation on the initial edge image areas corresponding to the two areas. If the position of a certain pixel in the two areas is both "1" or "0", it indicates that the judgment of whether the pixel is an edge in the two areas is consistent, and the position of the corresponding pixel in the difference map is recorded as "0"; if it is inconsistent, the position of the corresponding pixel is recorded as "1". Through the above process, the difference map E corresponding to the pixel and the first area to be fused can be obtained. j .

[0157] Based on the coordinates of any pixel with a value of 1 in the difference map, the corresponding pixel in the first area to be fused is marked as an undetermined edge point, such as Figure 5 The pixels marked with “*”, where the value is 1, are the edge points in the first area to be fused;

[0158] According to the region growing method, within a preset interval (such as 5×5), the number of edge points connected to the pending edge point is counted. When the number is greater than or equal to a preset threshold (such as 5), the pending edge point is recorded as a fused edge point. If the number of connected edge points is less than 5, the pending edge point is removed as a noise point. After that, the first region to be fused in the initial edge image is updated according to the marked fused edge points to generate a fused edge image.

[0159] Based on the fusion of edge image and sampling image, the disparity map is generated by using scale-invariant feature transformation algorithm.

[0160] In any of the above embodiments, further, in order to more accurately determine the spilled objects, the spillage determination unit 40 is configured as follows:

[0161] Fit the ground plane based on the disparity map;

[0162] When the motion trajectory of the second segmented target intersects with the fitted ground plane, or, based on a preset time interval, when the distance between the second segmented target and the ground plane decreases, and from a certain moment on, the distance is less than a preset distance threshold, the second segmented target is determined to be a spilled object.

[0163] In any of the above embodiments, further, in the process of determining whether a spilled object is a spilled object, the avoidance behavior of the following vehicle may be introduced, that is, if there is an avoidance behavior, the front object is considered to be a spilled object. Therefore, the spillage determination unit 40 is further configured as follows:

[0164] When it is determined that the size information of any first segmented object belongs to the vehicle size threshold interval, the first segmented object is recorded as a vehicle object;

[0165] Calculate the position information of the vehicle target. When it is determined that the position information of the vehicle target is behind the position information of the second segmented target, determine whether the vehicle target has an avoidance behavior. If so, mark the second segmented target as a spilled object. If not, mark the second segmented target as a non-spilled object.

[0166] Embodiment 3:

[0167] This embodiment further provides a computer-readable storage medium, in which a computer program is stored. When the stored computer program is executed by a processor, the real-time detection method for spilled objects based on a smart lamp pole of any one of the above-mentioned embodiments 1 is implemented.

[0168] Specifically, the computer program includes computer instructions, which are stored in a computer-readable storage medium; the processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method steps in any one of the above-mentioned embodiments 1.

[0169] The above-mentioned computer-readable storage medium may include random access memory (RAM), read-only memory (ROM), compact disc read-only memory (CD-ROM) or other optical disc storage, magnetic disk storage or other magnetic storage devices, etc.

[0170] The above-mentioned processor may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0171] So far, various embodiments of the present application have been described in detail. In order to avoid obscuring the concept of the present application, some details known in the art are not described. Based on the above description, those skilled in the art can fully understand how to implement the technical solution disclosed herein.

[0172] Although some specific embodiments of the present application have been described in detail through examples, those skilled in the art should understand that the above examples are only for illustration rather than for limiting the scope of the present application.

[0173] The steps in this application can be adjusted in order, combined, and deleted according to actual needs.

[0174] Although the present application is disclosed in detail with reference to the accompanying drawings, it should be understood that these descriptions are merely exemplary and are not intended to limit the application of the present application. The scope of protection of the present application is defined by the appended claims and may include various modifications, alterations and equivalents made to the invention without departing from the scope and spirit of the present application.

Claims

1. A real-time detection method for spilled objects based on smart lamp poles, characterized in that: The method comprises: Step 1: Perform target segmentation on the generated disparity map to generate the first segmentation target; Step 2: based on the image coordinates of the target frame of the first segmentation target and the corresponding disparity value, the size information of the first segmentation target is calculated, and the first segmentation target whose size information is smaller than the size threshold is recorded as the second segmentation target; Step 3: acquiring position information of the second segmentation target in the disparity map of the continuous multiple frames in time sequence, and fitting the motion trajectory of the second segmentation target based on the position information; Step 4: Based on the motion trajectory, determine whether the second segmented target is a spilled object.

2. The real-time detection method for spilled objects based on a smart lamp pole according to claim 1, characterized in that: The disparity map is generated by a binocular camera, and the binocular camera is installed on a smart light pole. A plurality of the smart light poles are arranged in sequence on both sides of the road. The step 1 specifically includes: Step 101: labeling the disparity maps generated by the multiple binocular cameras at the same time, wherein the labels are determined by the numbers of the smart lamp poles on which the binocular cameras are installed, and the numbers increase in sequence along the driving direction of the vehicle; Step 102: performing object segmentation on the labeled disparity map, and calculating the spatial position of each segmented third segmentation object in the world coordinate system; Step 103: When it is determined that the difference between the spatial positions of the third segmentation targets in the disparity images of two frames with adjacent labels is less than or equal to the screening threshold, the third segmentation targets in the disparity images with larger labels are filtered out, and the remaining third segmentation targets are recorded as the first segmentation targets.

3. The real-time detection method for spilled objects based on a smart lamp pole according to claim 1, characterized in that: Before step 1, the method further includes: Step 111: converting the acquired sampled image into a grayscale image, and filtering the grayscale image to generate a filtered image; Step 112: Calculate an adaptive threshold value for each pixel in the filtered image, wherein the adaptive threshold value is determined by the gray value of the pixel and its corresponding weight, and the weight is determined by the absolute value of the difference between the gray value of the pixel and the gray mean value in the corresponding pixel window; Step 113: Based on the adaptive threshold, a Sobel edge detection algorithm is used to generate an initial edge image, and based on the initial edge image and the sampled image, a scale-invariant feature transformation algorithm is used to generate the disparity map.

4. The real-time detection method for spilled objects based on a smart lamp pole according to claim 3, characterized in that: The filtering of the grayscale image to generate a filtered image specifically includes: Based on the preset filtering window, calculate the minimum grayscale value, maximum grayscale value and grayscale median corresponding to the current center point; When it is determined that the difference between the minimum grayscale value and the maximum grayscale value is less than the grayscale threshold, the grayscale value of the current center point is recorded as the center point grayscale value; When it is determined that the difference between the minimum grayscale value and the maximum grayscale value is greater than or equal to the grayscale threshold, determine whether the grayscale median is greater than the minimum grayscale value and less than the maximum grayscale value. If so, record the grayscale median as the grayscale value of the center point. If not, increase the size of the preset filter window according to the preset step size, and recalculate the minimum grayscale value, maximum grayscale value and grayscale median corresponding to the current center point. The filtered image is generated based on the grayscale value of the center point.

5. The real-time detection method for spilled objects based on a smart lamp pole according to claim 3, characterized in that: The disparity map is generated by a binocular camera, and the binocular camera is installed on a smart light pole. A plurality of the smart light poles are sequentially arranged on both sides of the road. The step 113 specifically includes: Calibrate the binocular cameras installed on two adjacent smart light poles to determine the overlapping area of ​​the field of view of the two adjacent binocular cameras; Based on the overlapping area of ​​the field of view, the area to be fused is determined in the initial edge images corresponding to the front and rear binocular cameras, respectively, wherein the area to be fused corresponding to the front binocular camera is recorded as the first area to be fused, and the area to be fused corresponding to the rear binocular camera is recorded as the second area to be fused; Calculating a difference map between the first region to be fused and the second region to be fused; Based on the coordinates of any pixel point with a value of 1 in the difference map, marking the corresponding pixel point in the first area to be fused as a pending edge point; According to the region growing method, within a preset interval, the number of edge points connected to the pending edge point is counted, and when it is determined that the number is greater than or equal to a preset threshold, the pending edge point is recorded as a fused edge point, and the first to-be-fused region in the initial edge image is updated according to the fused edge point to generate a fused edge image; Based on the fused edge image and the sampled image, a scale-invariant feature transformation algorithm is adopted to generate the disparity map.

6. The real-time detection method for spilled objects based on a smart lamp pole according to any one of claims 1 to 5, characterized in that: The step 4 specifically includes: Fitting a ground plane based on the disparity map; When the motion trajectory of the second segmented target intersects the fitting ground plane, or, According to a preset time interval, when the distance between the second segmentation target and the ground plane decreases, The second segmented target is determined to be a spilled object.

7. The real-time detection method for spilled objects based on a smart lamp pole according to any one of claims 1 to 5, characterized in that: The method further comprises: When it is determined that the size information of any of the first segmented objects belongs to the vehicle size threshold interval, the first segmented object is recorded as a vehicle object; calculating the position information of the vehicle target, and when it is determined that the position information of the vehicle target is located behind the position information of the second segmented target, determining whether the vehicle target has an avoidance behavior, If yes, mark the second segmented target as a spilled object. If not, the second segmented target is marked as a non-scattered object.

8. The real-time detection system for spilled objects based on smart lamp poles is characterized by: The system comprises: A first segmentation unit, wherein the first segmentation unit is configured to perform target segmentation on the generated disparity map to generate a first segmentation target; a second segmentation unit, the second segmentation unit being configured to calculate size information of the first segmentation target based on the image coordinates of the target frame of the first segmentation target and the corresponding disparity value, and record the first segmentation target whose size information is smaller than a size threshold as a second segmentation target; A trajectory fitting unit, wherein the trajectory fitting unit is configured to obtain position information of the second segmentation target in the disparity map of consecutive frames in a time sequence, and fit a motion trajectory of the second segmentation target based on the position information; A spillage determination unit is configured to determine whether the second segmented target is a spilled object based on the motion trajectory.

9. The real-time detection system for spilled objects based on smart lamp poles according to claim 8, characterized in that: The spillage determination unit is configured to: Fitting a ground plane based on the disparity map; When the motion trajectory of the second segmented target intersects the fitting ground plane, or, According to a preset time interval, when the distance between the second segmentation target and the ground plane decreases, The second segmented target is determined to be a spilled object.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images