Parking space state multi-task detection method based on look-around image

Abstract

The invention discloses a parking space state multi-task detection method based on a look-around image, and the method comprises the following steps: S1, collecting and preprocessing a vehicle look-around image; s2, marking the acquired image data and generating preprocessed data; s3, utilizing the preprocessed data to construct a multi-task detection model of the application parking space state; s4, the detection information is transmitted to the vehicle machine system and the application of the vehicle machine system; the method comprises a completion process from image and data preprocessing, model establishment and training to final use, on the basis of realizing two tasks of detecting the parking space and the target object at the same time, the detection results of the two tasks are combined, the parking space state information is perfected, and the detection accuracy of the parking space is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of vehicle automatic driving, in particular to a parking space state multi-task detection method based on surround view images. BACKGROUND

[0003] In the field of parking, the perception detection technology is mainly divided into two categories, namely the detection of parking spaces and the detection of surrounding targets. After years of development, the detection methods for parking spaces and targets have gradually expanded and introduced the use of sensors such as vision, with the aid of neural network parking detection methods and obstacle detection methods based on traditional ultrasonic rough positioning.

[0004] However, the above detection methods only focus on single task in parking space and surrounding environment detection, although corresponding improvements and optimizations have been made in detection accuracy, model lightweight, etc., but there is a lack of contact mechanism between each other, and the detection results cannot be mutually referenced in the actual execution process, and there is still room for improvement in restoring the actual parking scene. SUMMARY

[0005] The purpose of the present application is to provide a parking space state multi-task detection method based on surround view images to solve the problems raised in the above background.

[0006] To achieve the above purpose, the present application provides the following technical solutions: A parking space state multi-task detection method based on surround view images, comprising the following steps: S1: According to the use scenario, the surround view image of the vehicle is collected and preprocessed; S2: Labeling the collected image data and generating the preprocessed data; S3: Constructing a parking space state multi-task detection model, including an analysis backbone module, a feature fusion module, a prediction output module, an evaluation module and an information integration module; Based on the preprocessed data, the analysis backbone module, the feature fusion module, the prediction output module and the evaluation module in the parking space state multi-task detection model are used to train the model and judge the model state; If the model state identifier is "available", it means that the model has reached the expected function under the use scenario after training, otherwise, continue to train and judge until the model state identifier is "available"; In the model state is identified as "available" state, the parking space state multi-task detection model prediction output module output parking information, target object information, respectively saved in the parking space result container, target object result container, through the information integration module, using the detection content in the container, analyze the position corresponding relationship of all classes "parking mark point" target objects in u parking spaces and v target objects in the nth image, find the undetected parking information, then saved in the parking space result container, and update the number of parking spaces u new ; Analyze the position corresponding relationship of all classes "obstacle" target objects in u new parking spaces and v target objects updated in the corresponding surround view image, analyze the state of all parking spaces, and save in the parking space result container; S4: output the parking information of the updated parking space result container and the target object information of the retained target object result container, and deliver the detection information to the car system and its applications.

[0007] Preferably, in S1, the fisheye camera installed on the upper middle part of the front and rear license plates and the lower middle part of the left and right rearview mirrors is used to collect the images around the vehicle in real time. After distortion removal and perspective transformation of the images in each direction, the images are converted into overhead view images. Then, the overhead view images are spliced according to the corresponding regions, the gaps formed after splicing the overlapping regions are eliminated, and the complete surround view images are obtained after illumination homogenization.

[0008] Preferably, in S2, the surround view images without processing are defined as "actual images", the subscript n represents the nth surround view image, the superscript H represents the height value, the superscript W represents the width value, and the superscript C represents the channel value. The "actual images" are randomly sampled at a certain proportion to obtain "images to be labeled"; The parking spaces in the "images to be labeled" are labeled, and the labeling content includes parking space classification information , which is divided into "parallel parking space", "perpendicular parking space", "diagonal parking space", and two parking space entrance point coordinates , , , , where the superscript p represents the parking space information, the superscript t represents the true value, the subscript u represents the uth parking space in the nth image, and x1, y1 and x2, y2 represent the horizontal and vertical coordinate values of the two entrance points of the parking space in the image; The target objects in the "images to be labeled" are labeled, and the labeling content includes target object classification information , which is divided into "parking mark point" and "obstacle", and two detection box coordinates , ), ( , ), wherein the superscript o represents the target object information, the subscript v represents the vth target object in the nth image, xa, ya, and xb, yb represent the horizontal and vertical coordinate values of the top-left corner and the bottom-right corner of the target object detection frame in the image; After the labeling is completed, the "to-be-labeled image" containing the labeled content, the corresponding parking space label, and the target object label are defined as "preprocessed data", and the surround view image corresponding to the "preprocessed data" is deleted from the "actual image".

[0009] Preferably, the parking space state multi-task detection model is constructed by the following steps: S31: constructing an analysis backbone module for format conversion of the input image; S32: constructing a feature fusion module for format conversion and feature fusion of the output of the analysis backbone module; S33: constructing a prediction output module for parsing the output content of the feature fusion module, extracting the description information of the parking space and the target object, and outputting the specific value; S34: constructing an evaluation module to evaluate the model state; S35: constructing an information integration module including a parking space result container, a target object result container, and an analysis and judgment unit.

[0010] Preferably, based on the preprocessed data, the analysis backbone module, the feature fusion module, the prediction output module, and the evaluation module in the parking space state multi-task detection model are used to train the model and judge the model state, including the following steps: SS1: setting the initial value of the model state identifier to "to be trained" and setting the maximum number of training N; SS2: if the model state identifier is "to be trained", using the "preprocessed data" as the input source of the model, executing SS4, otherwise, executing SS3; SS3: if the model state identifier is "available", using the "actual image" as the input source of the model, executing SS4; SS4: using the analysis backbone module of the parking space state multi-task detection model to process the surround view image in the input source , using the convolution unit CBT j and the feature enhancement unit SFC k of the analysis backbone module to proportionally reduce the height and width of the surround view image, reduce the calculation amount, improve the running efficiency, increase the number of channels, extract and superimpose the parking space information and obstacle information in the image, and then refine and save them in different channels, and finally output the data container containing the size and channels after processing by the analysis backbone module, i.e., the tensor ; SS5: using the feature fusion module to process the tensor output by the analysis backbone module Further classification of features: using the pooling unit MC of the feature fusion module q tensor According to different pooling kernel sizes, the tensor reflecting different features at different positions is obtained by performing a pooling operation , where subscript q represents using the qth pooling unit for processing; The tensor output by the analysis backbone module , together with the tensor output by the pooling unit , is input into the feature aggregation unit MBC of the feature fusion module r The size values of these tensors are averaged to obtain the output size value, and a uniform transformation is performed, and the channels of these tensors are accumulated to obtain the output tensor ; SS6: using the prediction output module to obtain the detection results of the parking space and the target object by analyzing the output tensor of the feature fusion module : Using the parking space detection unit CPP e The channels of the output tensor are transformed and analyzed to finally obtain six prediction values, including confidence , parking space classification information , and the coordinates of the two parking space entry points in the tensor size , ), , , where the confidence value is 0~1, and the closer to 1, the closer to the true value. The size ratio difference between the surround view image and the output tensor is calculated, and the two parking space entry point coordinates are proportionally converted to the corresponding coordinate values on the surround view image. The converted coordinate values, together with the confidence and classification information, are output as the parking space information output content of the prediction output module according to the corresponding parking space number; Using the target object detection unit COO f The channels of the output tensor are transformed and analyzed to finally obtain six prediction values, including confidence , target object classification information , and two detection frame coordinates , ), , , the surround view image and the output tensor the size ratio difference, the coordinates of the upper left corner point and the lower right corner point of the target object detection box are proportionally converted to the coordinate values on the corresponding surround view image, and the converted coordinate values are output as target object information of the prediction output module together with the confidence and classification information according to the number of the corresponding target object; SS7: If the model state identifier is "available", the parking space information and the target object information output by the prediction output module are saved in the parking space result container and the target object result container respectively, otherwise, SS8 is executed; SS8: The bias calculation unit of the evaluation module is used to calculate the total bias of the output content of the prediction output module and the corresponding labeled true value in the "preprocessing data" by formula (1) : (1) In formula (1), the symbol represents whether the parking space category or the target object category matches the true value, and is set to 1 if yes, otherwise 0, represents the parking space bias, represents the target object bias; SS9: The model state judgment unit of the evaluation module is used to judge the model state, and judge whether the total bias is less than the preset threshold β or whether the specified training number N is reached, if yes, the model state identifier is modified to "available", and SS3 is executed, otherwise, the model state identifier is kept as "to be trained", and SS2 is executed.

[0011] Preferably, the information integration module is used to analyze the position corresponding relationship of all target objects with the category of "parking space marker point" in the u parking spaces and the v target objects in the nth image, find the undetected parking space information, and then save it in the parking space result container and update the number of parking spaces u new , comprising the following steps: According to the rule "whether the parking space entry point coordinates are within the detection box range of the target object with the category of "parking space marker point", the corresponding relationship between the u x 2 parking space entry points contained in the u parking spaces and all "parking space marker point" target objects is judged one by one; If all correspondences are found, the number of parking spaces u is set to the number of parking spaces u new of the updated parking space result container of the corresponding surround view image, otherwise, the target object with the category of "parking space marker point" which does not find the corresponding parking space entry point in the corresponding surround view image is used to find the adjacent parking space entry point, generate a new parking space and save it in the parking space result container, and the number of parking spaces u is added to the number of newly added parking spaces, and set to the number of parking spaces u new of the updated parking space result container of the corresponding surround view image.

[0012] Preferably, the updated unew The position corresponding relationship between each parking space and all the target objects of the category "obstacle" in the v target objects is analyzed, and the state of all the parking spaces is saved in the parking space result container, including the following steps: The u new The length of the line connecting the two parking space entry points of the u new The parking space size is completed according to the parking space size setting value corresponding to the parking space classification information, and the parking space area is drawn one by one in the image; The overlap degree of each parking space area of the u new If the overlap degree is greater than the overlap degree threshold γ, the parking space state is defined as "occupied" state, otherwise, it is defined as "vacant" state, and the parking space state is saved in the corresponding parking space information in the parking space result container.

[0013] Preferably, the target objects of the category "parking space marker point" not found in the corresponding surround view image are used to find the adjacent parking space entry point, generate a new parking space and save it in the parking space result container, including: N1: Calculate the center point coordinates of the bounding box of the w target objects of the category "parking space marker point" not found in the corresponding parking space entry point in the v target objects one by one using formula (2): , ): (2) In formula (2), subscript w represents the "parking space marker point" target object box in the v target objects not found in the corresponding parking space entry point; N2: Calculate the distance between the center point coordinates and the parking space entry point coordinates of all parking spaces one by one using formula (3): , : (3) In formula (3), D is the distance between two points, and respectively represent the horizontal and vertical coordinate values of the parking space entry point whose distance needs to be calculated; Take the minimum value among all the calculated distances to get the minimum distance D , of the to-be-determined parking space corresponding to all the center point coordinates w ; N3: Judge all the to-be-determined parking space minimum distances D w and update the parking space information: Find the parking space point corresponding to D w , compare the set range value of the parking space entry line in the corresponding category parking space size setting value with D w , and judge D wwhether in the range; if D w in the range, it is judged that a new parking space is found, the confidence of the new parking space is set to 1, and the parking space category of the new parking space is set to D w The two parking space entry point coordinates of the new parking space are set to D w The corresponding parking space point coordinates and the corresponding center point coordinates( , ), the parking space information of the new parking space is saved in the parking space result container, and the number of parking spaces in the container is increased by one, otherwise, it is judged that a new parking space is not found.

[0014] The beneficial effects of the present application are as follows: 1. The present application proposes a parking space state multi-task detection method based on surround view images, including the complete process from image and data preprocessing, model establishment and training to final use, on the basis of realizing the detection of two tasks of parking space and target object, combining the detection results of the two tasks, the parking space state information is improved, and the detection accuracy of the parking space is improved.

[0015] 2. The present application proposes a parking space state multi-task detection model design method, which can be developed by self-defined combination based on existing neural network modules, simplifying the design process, and at the same time, corresponding specification model design can be carried out according to the hardware conditions of the application scene, improving the generality of the model.

[0016] 3. The present application proposes a method for analyzing parking space state by comprehensively using parking space and target object detection results, by using the comparison of target object and parking space detection results in spatial position, according to the category of different target objects, the missed detection parking space is searched and filled, and the parking space state information is judged, realizing the complementary use of multi-task detection information. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 The flow chart of the parking space state multi-task detection method of the present application; Figure 2 The parking space state multi-task detection model structure diagram of the present application; Figure 3 The parking space and target object visualization output effect diagram of the present application. DETAILED DESCRIPTION

[0018] The technical solutions of the present application will be further described in detail below in combination with the drawings and specific embodiments of the present application.

[0019] In this embodiment, as shown in the figure, a parking space state multi-task detection method based on surround view images includes the following steps: Figure 1 ​S1: Vehicle surround view image acquisition and preprocessing, through the fisheye camera installed on the front, rear license plate middle top, and left and right rearview mirror lower middle position, before putting into use, using the calibration board around the vehicle, taking Zhang Zhengyou calibration method to carry out joint calibration of fisheye camera in front, rear, left and right directions, to ensure image effect, real-time acquisition of vehicle surrounding image, distortion removal and perspective transformation of image in each direction to convert to overhead view image, then the overhead view image is spliced according to the corresponding area, the gap formed after splicing the overlapping area is eliminated, such as using median filtering method or weighted average fusion method, then the light uniformization is carried out, such as using gamma correction or histogram equalization processing, to eliminate the color difference between each direction area of the surround view image caused by light difference, then the light uniformization is carried out to obtain the complete surround view image; Wherein, Zhang Zhengyou calibration method is a classic method for camera calibration proposed by Dr. Zhang Zhengyou of Microsoft Asia Research Institute, which has a wide application in computer vision field, and its core function is to determine the internal parameters (such as focal length, principal point position, etc.) and external parameters (such as rotation matrix, translation vector) of the camera, so as to establish the corresponding relationship between image pixel coordinates and three-dimensional space point coordinates; S2: Labeling the collected image data and generating preprocessed data: The surround view image without any processing is defined as "actual image", the subscript n represents the nth surround view image, the superscript H represents the height value, the superscript W represents the width value, and the superscript C represents the channel value. After a certain proportion of random sampling on the surround view image in the "actual image", the default proportion is 40%, the "to-be-labeled image" is obtained. The parking space in the "to-be-labeled image" is labeled, and the labeling software provided by the open source DMPR parking space detection model can be used. The labeling content includes parking space classification information , which is divided into "parallel parking space", "vertical parking space" and "diagonal parking space", as well as two parking space entrance point coordinates , , , , wherein the superscript p represents the parking space information, the superscript t represents the true value, the subscript u represents the nth image, and x1, y1 and x2, y2 represent the horizontal and vertical coordinate values of the two entrance points of the parking space in the image. The target object in the "to-be-labeled image" is labeled, and the open source labelimg labeling frame software is used. The labeling content includes target object classification information , which is divided into "parking space marker" and "obstacle", as well as two detection frame coordinates , , ,​ ), wherein the superscript o represents the target object information, the subscript v represents the vth target object in the nth image, xa, ya, and xb, yb represent the horizontal and vertical coordinate values of the top-left corner and bottom-right corner of the target object detection frame in the image; After the labeling is completed, the labeled "to-be-labeled image" containing the corresponding parking space label and target object label is defined as "preprocessed data". On this basis, the "preprocessed data" can be further divided into a training verification set for training the model and a test set for testing the model according to a ratio of 9:1. The training verification set is further divided into a training set and a test set according to a ratio of 9:1, and the surround view image corresponding to the "preprocessed data" is deleted from the "actual image"; S3: constructing a parking space state multi-task detection model, including an analysis backbone module, a feature fusion module, a prediction output module, an evaluation module, and an information integration module. The above modules can be constructed using templates provided in software libraries for neural network programming development, such as pytorch or tensorflow. Based on the preprocessed data, the analysis backbone module, the feature fusion module, the prediction output module, and the evaluation module in the parking space state multi-task detection model are used to train the model and determine the model state. If the model state is identified as "available", it means that the model has reached the expected function in the use scenario after training. Otherwise, the training continues until the model state is identified as "available". When the model state is identified as "available", the parking space information and the target object information output by the prediction output module of the parking space state multi-task detection model are saved in the parking space result container and the target object result container, respectively. Using the detection content in the containers, the information integration module analyzes the position correspondence relationship of all target objects with the class "parking space marker" in the u parking spaces and the v target objects in the nth image, finds the undetected parking space information, and then saves it in the parking space result container and updates the number of parking spaces u new ; The analysis of the position correspondence relationship of all target objects with the class "obstacle" in the u new updated parking spaces and the v target objects in the corresponding surround view image analyzes the state of all parking spaces and saves them in the parking space result container. S4: outputting the parking space information of the updated parking space result container and the target object information of the retained target object result container, and delivering the detection information to the car system and its applications for visual display and subsequent parking control. The obstacle detection frame region in the target object detection result is used for obstacle avoidance operation in parking control, and the middle point between the two parking entrance points can be used as a reference point in the parking process, corresponding to the center of the vehicle rear axle, to ensure accurate parking in the parking space. The visual display is as follows Figure 3As shown, the related development can be based on the OpenCV software library, and the target object detection result is represented by the frame in the surround view image. The target object category name is attached to the text above the frame, i.e., the text "marking_point" and "obstacle" above the frame represent "parking space marking point" and "obstacle", and the number behind the text is the detection confidence value. The closer to 1, the higher the confidence. The green circle position in the figure is the parking space entrance point. When the parking space is used, a straight line is used to connect the parking space entrance point, and two parking space depth lines are drawn according to the parking space size setting value, which together constitute a parking space. The number attached to the parking space represents the parking space confidence, i.e., the large number "1.00" in the figure. The closer to 1, the higher the confidence.

[0020] In this embodiment, the S3 constructs a parking space state multi-task detection model, which includes the following steps: Figure 2 As shown, the related development can be based on the OpenCV software library, and the target object detection result is represented by the frame in the surround view image. The target object category name is attached to the text above the frame, i.e., the text "marking_point" and "obstacle" above the frame represent "parking space marking point" and "obstacle", and the number behind the text is the detection confidence value. The closer to 1, the higher the confidence. The green circle position in the figure is the parking space entrance point. When the parking space is used, a straight line is used to connect the parking space entrance point, and two parking space depth lines are drawn according to the parking space size setting value, which together constitute a parking space. The number attached to the parking space represents the parking space confidence, i.e., the large number "1.00" in the figure. The closer to 1, the higher the confidence. S31: Construct an analysis backbone module for format conversion of the input image; S32: Construct a feature fusion module for format conversion and feature fusion of the output of the analysis backbone module; S33: Construct a prediction output module for analyzing the output content of the feature fusion module, extracting the description information of the parking space and the target object, and outputting specific values; S34: Construct an evaluation module to evaluate the model state; S35: Construct an information integration module, including a parking space result container, a target object result container, and an analysis and judgment unit.

[0021] In this embodiment, the S3 constructs a parking space state multi-task detection model, which includes the following steps: SS1: Set the initial value of the model state identifier to "to be trained", and set the maximum number of training N; SS2: If the model state identifier is "to be trained", use the model input source as "preprocessed data", execute SS4, otherwise, execute SS3; SS3: If the model state identifier is "available", use the model input source as "actual image", execute SS4; SS4: Use the analysis backbone module of the parking space state multi-task detection model to process the surround view image in the input source , use the convolution unit CBT j and the feature enhancement unit SFC k, and width of the surround view image are scaled down proportionally to reduce the amount of calculation while improving the running efficiency, the number of channels is increased, the parking space information and obstacle information contained in the image are extracted and superimposed, and then refined and saved in different channels, and finally the data container containing the size and channel after the processing of the analysis backbone module is output, that is, the tensor ; SS5: using the feature fusion module to further classify the tensor output by the analysis backbone module: The pooling unit MC of the feature fusion module q performs pooling operation on the tensor according to different pooling kernel sizes to obtain tensors reflecting different features at different positions , where subscript q represents processing using the qth pooling unit; The tensor output by the analysis backbone module is input into the feature aggregation unit MBC of the feature fusion module together with the tensor output by the pooling unit r , the size values of these tensors are averaged to obtain the output size value, and a unified transformation is performed, and the channels of these tensors are accumulated to obtain the output tensor .

[0022] SS6: using the prediction output module to obtain the detection results of the parking space and the target object by analyzing the output tensor of the feature fusion module : The parking space detection unit CPP e performs transformation analysis on the channels of the output tensor to finally obtain six prediction values, including confidence , parking space classification information , and the coordinates of two parking space entry points in the size of the tensor , ), ( , ), wherein the confidence value is 0~1, and the closer to 1, the closer to the true value, the size ratio difference between the surround view image and the output tensor is calculated, the two parking space entry point coordinates are proportionally converted to the coordinate values on the corresponding surround view image, and the converted coordinate values are output as the parking space information output content of the prediction output module together with the confidence, classification information, and the number of corresponding parking spaces; The target object detection unit COO f performs transformation analysis on the channels of the output tensor to finally obtain six prediction values, including confidence , target object classification information And the coordinates of the two detection boxes ( , ), ( , ), calculate the toroidal image and output tensor The size ratio difference is used to proportionally convert the coordinates of the upper left and lower right corners of the target object detection box to the corresponding coordinate values ​​on the surrounding view image. The converted coordinate values, along with the confidence level and classification information, are used as the target object information output content of the prediction output module according to the corresponding target object number.

[0023] SS7: Determine if the model status indicator is "available". If yes, save the parking space information and target object information output by the prediction output module into the parking space result container and the target object result container, respectively. Otherwise, execute SS8. SS8: Using the deviation calculation unit of the evaluation module, calculate the total deviation between the output of the prediction output module and the corresponding true value marked in the "preprocessed data" using equation (1). : (1) In formula (1), the symbol This indicates whether the parking space category or target object category matches the actual value. If yes, set it to 1; otherwise, set it to 0. Indicates parking space deviation. Indicates the deviation from the target object; SS9: Use the model state judgment unit of the evaluation module to judge the model state and determine the total deviation. If the model status is less than the preset threshold β or the specified number of training iterations N has been reached, change the model status to "available" and execute SS3; otherwise, keep the model status as "to be trained" and execute SS2.

[0024] In this embodiment, in step S3, the information integration module uses the detected content in the container to analyze the positional correspondence of all targets classified as "parking space markers" among u parking spaces and v targets in the nth image, finds undetected parking space information, saves it in the parking space result container, and updates the number of parking spaces u. new This includes the following steps: Based on the rule of "whether the coordinates of the parking space entrance point are within the detection frame of the target object of the category "parking space marker", the correspondence between the u×2 parking space entrance points contained in u parking spaces and all "parking space marker" target objects is determined one by one; If all correspondences are correct, set the number of parking spaces u to the number of parking spaces u in the container of the updated parking space result of the corresponding surround view image. new, otherwise, using the target objects of the category "parking space marker point" that are not found in the corresponding loop view image, find the adjacent parking space entrance points, generate new parking spaces and save them in the parking space result container, add the number of new parking spaces to the number of parking spaces u, and set the number of parking spaces u of the parking space result container corresponding to the updated loop view image new .

[0025] In this embodiment, the S3 analyzes the position correspondence relationship between the updated u new parking spaces in the corresponding loop view image and all target objects of the category "obstacle" in v target objects, analyzes the state of all parking spaces, and saves them in the parking space result container, including the following steps: Calculate the length of the line connecting the two parking space entrance points of the u new parking spaces, complete the parking space size according to the parking space size setting value corresponding to the parking space classification information, and draw the parking space area one by one in the image; For each parking space area of the u new parking spaces, compare the overlap degree with the detection box range of all target objects of the category "obstacle", if it is greater than the overlap degree threshold γ, define the parking space state as "occupied" state, otherwise, define it as "vacant" state, and save the parking space state in the corresponding parking space information in the parking space result container.

[0026] In this embodiment, the S3 uses the target objects of the category "parking space marker point" that are not found in the corresponding loop view image to find the adjacent parking space entrance points, generate new parking spaces and save them in the parking space result container, including: N1: Calculate the center point coordinates of the bounding box of the w target objects of the category "parking space marker point" that are not found in the corresponding parking space entrance points one by one using formula (2) , The center point is used because there are cases where "parking space marker points" are mostly at the edge of the image or are blocked, resulting in only a small part of the features: (2) In formula (2), subscript w represents the "parking space marker point" target object bounding box of the w target objects that are not found in the corresponding parking space entrance points in the v target objects; N2: Calculate the distance between the center point coordinates , and the parking space entrance point coordinates of all parking spaces one by one using formula (3): (3) In formula (3), D is the distance between two points, and respectively represent the horizontal and vertical coordinate values of the parking space entrance point whose distance needs to be calculated; The minimum value among all calculated distances is used to obtain the coordinates of all center points. , The minimum distance D corresponding to the undetermined parking space w ; N3: Determine the minimum distance Dw for all undetermined parking spaces and update the parking space information: Find D w The corresponding parking space belongs to a specific parking space category. This is determined by the setting range of the parking space entrance line within the corresponding category's parking space size settings, and then compared to D. w Compare and determine D w Is it within range? If D w If the location is within the specified range, a new parking space is found. The confidence level for this new parking space is set to 1. Since the model uses the detection results from both tasks for comprehensive reasoning, the confidence level is uniformly set to 1. The parking space category of the new space is designated as D. w The corresponding parking space entrance point is located in the parking space category. In reality, adjacent parking spaces are of the same category. The coordinates of the two parking space entrance points of the new parking space are set as D. w The corresponding parking spot coordinates and the corresponding center point coordinates ( , The new parking space information is saved in the parking space result container, and the number of parking spaces in the container is incremented by one. Otherwise, it is determined that no new parking space has been found.

[0027] In this embodiment, an electronic device, such as a memory and a processor, is also required. The memory is used to store a program that supports the processor in executing the multi-task detection method for parking space status, and the processor is configured to execute the program stored in the memory.

[0028] In this embodiment, a computer-readable storage medium is also required, on which a computer program is stored. When the computer program is run by the processor, it executes the steps of the parking space status multi-task detection method.

[0029] The embodiments described above merely illustrate implementation methods of the present invention and should not be construed as limiting the scope of the invention patent, nor as imposing any form of limitation on the structure of the present invention. It should be noted that those skilled in the art can make various changes and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention.

Claims

1. A multi-task detection method for parking space status based on surround view images, characterized in that: Includes the following steps: S1: Collect and preprocess vehicle surround view images according to the usage scenario; S2: Annotate the acquired image data and generate preprocessed data; S3: Construct a multi-task detection model for parking space status, including an analysis backbone module, a feature fusion module, a prediction output module, an evaluation module, and an information integration module; Based on preprocessed data, the analysis backbone module, feature fusion module, prediction output module and evaluation module in the parking space status multi-task detection model are used to train the model and determine the model status. If the model status is marked as "available", it means that the model has been trained to achieve the expected function in the usage scenario; otherwise, continue training and make judgments until the model status is marked as "available". When the model status is marked as "available," the parking space information and target object information output by the prediction output module of the multi-task parking space detection model are stored in the parking space result container and the target object result container, respectively. Through the information integration module, the detected content in the containers is used to analyze the positional correspondence of all targets of the category "parking space marker" in the u parking spaces and v target objects in the nth image, to find undetected parking space information, which is then stored in the parking space result container, and the number of parking spaces u is updated. new ; Analyze the updated u within the corresponding ring view image new The system identifies the positional correspondence between each parking space and all objects of the "obstacle" category among v target objects, analyzes the status of all parking spaces, and saves the results in a parking space container. S4: Output the updated parking space result container's parking space information and the retained target object result container's target object information, and send the detection information to the vehicle system and its applications.

2. The multi-task detection method for parking space status based on surround view images according to claim 1, characterized in that: In S1, fisheye cameras installed above the center of the front and rear license plates and in the middle of the lower part of the left and right rearview mirrors are used to collect images of the vehicle's surroundings in real time. The images from each direction are then converted into top-view images after distortion removal and perspective transformation. The top-view images are then stitched together according to the corresponding areas. Gaps formed after stitching together overlapping areas are eliminated. Finally, after illumination homogenization, a complete surround view image is obtained.

3. The multi-task detection method for parking space status based on surround view images according to claim 1, characterized in that: In S2, the unprocessed panoramic image is... Defined as "actual image", the subscript n represents the nth panorama image, the superscript H represents the height value, the superscript W represents the width value, and the superscript C represents the channel value. The "image to be labeled" is obtained by randomly sampling the panorama images in the "actual image" at a certain ratio. Label the parking spaces in the "image to be labeled", including parking space classification information. The parking space classification information is divided into "parallel parking spaces", "perpendicular parking spaces", and "angled parking spaces", as well as the coordinates of the two parking space entrance points ( , ), ( , ), where the superscript p indicates that it belongs to parking space information, the superscript t indicates the actual value, the subscript u indicates the u-th parking space in the n-th image, and x1, y1 and x2, y2 represent the horizontal and vertical coordinates of the two entrance points of the parking space in the image; Label the target objects in the "image to be labeled", including the classification information of the target objects. The target object classification information is divided into "parking space markers" and "obstacles", as well as the coordinates of two detection boxes ( , ), ( , ), where the superscript o indicates that it belongs to the target object information, the subscript v indicates the v-th target object in the n-th image, and xa, ya and xb, yb represent the horizontal and vertical coordinates of the upper left and lower right corners of the target object detection box in the image; After the annotation is completed, the content containing the annotated "image to be annotated" along with the corresponding parking space annotations and target object annotations is defined as "preprocessed data", and the panoramic image corresponding to the "preprocessed data" is deleted from the "actual image".

4. The multi-task detection method for parking space status based on surround view images according to claim 1, characterized in that: The construction of the multi-task detection model for parking space status includes the following steps: S31: Construct the main analysis module to perform format transformation on the input image; S32: Construct a feature fusion module to perform format transformation and feature fusion on the output of the main analysis module; S33: Construct a prediction output module to parse the output of the feature fusion module, extract descriptive information of parking spaces and target objects, and output specific values; S34: Construct the evaluation module to evaluate the model status; S35: Construct an information integration module, which includes a parking space result container, a target object result container, and an analysis and judgment unit.

5. The multi-task detection method for parking space status based on surround view images according to claim 1, characterized in that: Based on preprocessed data, the analysis backbone module, feature fusion module, prediction output module, and evaluation module of the parking space status multi-task detection model are used to train the model and determine the model status, including the following steps: SS1: Set the model status flag to "Waiting to Train" initially, and set the maximum number of training iterations N; SS2: If the model status is "To be trained", use "Preprocessed data" as the model input source and execute SS4; otherwise, execute SS3. SS3: If the model status is "available", use "actual image" as the model input source and execute SS4; SS4: The core analysis module of the multi-task detection model for parking space status, processing the surround view images from the input source. Using the convolutional units CBT of the analysis backbone module j and Feature Enhancement Unit (SFC) k The system proportionally reduces the height and width of the panoramic image, improving efficiency while reducing computational load. Simultaneously, it increases the number of channels, extracting and overlaying parking and obstacle information from the image, then refining and saving it into different channels. The final output is a tensor, a data container containing dimensions and channels, processed by the main analysis module. ; SS5: Using the feature fusion module to analyze the tensors output by the main analysis module. Further classification of features: Pooling unit MC of the feature fusion module q tensor Pooling operations are performed based on different pooling kernel sizes to obtain tensors that reflect different features at different locations. The subscript q indicates that the q-th pooling unit is used for processing; The tensors output by the analysis backbone module will be analyzed. Along with the tensors output by the pooling unit The feature aggregation unit (MBC) of the input feature fusion module r The output dimension value is calculated by averaging the dimension values ​​of these tensors, followed by a uniform transformation. Finally, the channels of these tensors are summed to obtain the output tensor. ; SS6: Using the prediction output module, the output tensor of the feature fusion module is parsed. Obtain the detection results for parking spaces and target objects: Using parking space detection unit CPP e Output tensor After transformation analysis, the channels ultimately yielded six predicted values, including confidence levels. Parking space classification information And the coordinates of the two parking space entrance points under the tensor dimension ( , ), ( , The confidence level is 0 to 1, with values ​​closer to 1 considered closer to the true value. The surrounding image is calculated. and output tensor The size ratio difference is used to proportionally convert the coordinates of the two parking space entrance points to the corresponding coordinate values ​​on the surrounding view image. The converted coordinate values, along with the confidence level and classification information, are used as the parking space information output content of the prediction output module according to the corresponding parking space number. Using the target object detection unit COO f Output tensor After transformation analysis, the channels ultimately yielded six predicted values, including confidence levels. Target object classification information And the coordinates of the two detection boxes ( , ), ( , ), calculate the toroidal image and output tensor The size ratio difference is used to proportionally convert the coordinates of the upper left and lower right corners of the target object detection box to the corresponding coordinate values ​​on the surrounding view image. The converted coordinate values, along with the confidence level and classification information, are used as the target object information output content of the prediction output module according to the corresponding target object number. SS7: Determine if the model status flag is "available". If yes, save the parking space information and target object information output by the prediction output module into the parking space result container and the target object result container, respectively. Otherwise, execute SS8. SS8: Using the deviation calculation unit of the evaluation module, calculate the total deviation between the output of the prediction output module and the corresponding true value in the "preprocessed data" using equation (1). : (1) In formula (1), the symbol This indicates whether the parking space category or target object category matches the actual value. If yes, set it to 1; otherwise, set it to 0. Indicates parking space deviation. Indicates the deviation from the target object; SS9: Use the model state judgment unit of the evaluation module to judge the model state and determine the total deviation. If the model status is less than the preset threshold β or the specified number of training iterations N has been reached, modify the model status flag to "available" and execute SS3; otherwise, keep the model status flag as "to be trained" and execute SS2.

6. The multi-task detection method for parking space status based on surround view images according to claim 1, characterized in that: The information integration module uses the detected content in the container to analyze the positional correspondence of all objects classified as "parking space markers" among u parking spaces and v objects in the nth image, searching for undetected parking space information, which is then saved in the parking space result container, and the number of parking spaces u is updated. new This includes the following steps: Based on the rule of "whether the coordinates of the parking space entrance point are within the detection box of the target object of the category "parking space marker", the correspondence between the u×2 parking space entrance points contained in u parking spaces and all "parking space marker" target objects is determined one by one; If all correspondences are correct, set the number of parking spaces u to the number of parking spaces u in the container of the updated parking space result of the corresponding surround view image. new Otherwise, using the target object of category "Parking Space Marker" within the corresponding surround view image where no corresponding parking space entrance point is found, search for nearby parking space entrance points, generate new parking spaces and save them in the parking space result container, add the number of newly added parking spaces to the number of parking spaces u in the updated parking space result container of the corresponding surround view image, and set it to the number of parking spaces u in the updated parking space result container of the corresponding surround view image. new .

7. The multi-task detection method for parking space status based on surround view images according to claim 1, characterized in that: Analyze the updated u within the corresponding ring view image new The system identifies the location correspondence between each parking space and all objects of the "obstacle" category among v target objects. It analyzes the status of all parking spaces and saves the results in a parking space result container. This includes the following steps: Calculate u new The length of the line connecting the two entrance points of each parking space is used to complete the parking space size based on the parking space size setting value of the corresponding parking space classification information, and the parking space area is drawn one by one in the image. For u new For each parking space area, the overlap is compared with the detection bounding box range of all targets classified as "obstacles". If the overlap is greater than the overlap threshold γ, the parking space status is defined as "occupied"; otherwise, it is defined as "vacant". The parking space status is then saved in the corresponding parking space information in the parking space result container.

8. The multi-task detection method for parking space status based on surround view images according to claim 6, characterized in that: Using the target object of category "Parking Space Marker" within the corresponding surround view image where no corresponding parking space entrance point was found, search for nearby parking space entrance points, generate new parking spaces, and save them in the parking space result container, including: N1: Using equation (2), calculate the center point coordinates of the target frames of w targets whose corresponding parking space entrance points were not found, which are classified as "parking space markers", one by one among the v target objects. , ): (2) In equation (2), the subscript w represents the target box of w "parking space markers" among v target objects that did not find the corresponding parking space entrance point; N2: Using equation (3), calculate the coordinates of the center point one by one. , Distance to the coordinates of the entrance points of all parking spaces: (3) In equation (3), D is the distance between the two points. and These represent the horizontal and vertical coordinates of the parking space entrance point for which the distance needs to be calculated; The minimum value among all calculated distances is used to obtain the coordinates of all center points. , The minimum distance D corresponding to the undetermined parking space w ; N3: Minimum distance D for all undetermined parking spaces w Make a judgment and update the parking space information: Find D w The corresponding parking space belongs to a specific parking space category. This is determined by the setting range of the parking space entrance line within the corresponding category's parking space size settings, and then compared to D. w Compare and determine D w Is it within range? If D w If the location is within the specified range, determine that a new parking space has been found, set the confidence level of the new parking space to 1, and classify the new parking space as D. w The corresponding parking space entrance point is located in the parking space category. The coordinates of the two parking space entrance points of the new parking space are set as D. w The corresponding parking spot coordinates and the corresponding center point coordinates ( , The new parking space information is saved in the parking space result container, and the number of parking spaces in the container is incremented by one. Otherwise, it is determined that no new parking space has been found.

Images