Vehicle key point location detection methods, devices, electronic equipment and storage media

Abstract

This application relates to a method, apparatus, electronic device, and storage medium for detecting the location of key points on a vehicle. The method may include: acquiring an image of a target vehicle in front of the vehicle as an image to be identified; inputting the image to be identified into a key point information extraction model, performing predictive processing on the key point information of the target vehicle to obtain the vehicle center point, multiple vehicle key points, and global and local offset information corresponding to each vehicle key point; using the four branches of information obtained from the above predictive processing to obtain the first and second position information of each vehicle key point; and then, based on the second position information and local offset information of each vehicle key point, obtaining the target position information of each vehicle key point. According to the technical solution provided in this application, the efficiency and accuracy of vehicle key point location detection can be improved.

Description

Technical Field

[0001] This application relates to the field of autonomous driving, and in particular to a method, device, electronic device, and storage medium for detecting the location of key points of a vehicle. Background Technology

[0002] With the rapid development of the autonomous driving industry, in order to ensure the safety of vehicles during operation, autonomous driving systems need to constantly monitor the driving conditions of other vehicles on the road. Therefore, the detection of key points on the vehicle is particularly important, and as a crucial branch of vehicle detection, it has received widespread attention in autonomous driving technology.

[0003] Traditional vehicle keypoint detection methods rely on a combination of highly specialized, handcrafted features and heuristics, followed by post-processing techniques. This approach results in high computational costs and divides the detection process into two parts, leading to lengthy detection times and preventing end-to-end detection. Furthermore, since vehicles in captured images are always partially occluded, some keypoints are obscured, impacting the algorithm's scalability. Summary of the Invention

[0004] This application provides a method, apparatus, electronic device, and storage medium for detecting the location of key points in a vehicle, to at least address the problem of improving the accuracy of key point locations in related technologies. The technical solution of this application is as follows:

[0005] According to a first aspect of the embodiments of this application, a method for detecting the location of key points of a vehicle is provided, comprising:

[0006] Acquire an image of the target vehicle in front of the vehicle as the image to be identified;

[0007] The image to be identified is input into the key point information extraction model, and the key point information of the target vehicle is predicted and processed to obtain the vehicle center point, multiple vehicle key points, and global offset information and local offset information corresponding to each vehicle key point. The global offset information represents the offset of each key point from the vehicle center point, and the local offset information represents the offset between vehicle key points with a preset adjacent relationship.

[0008] If the center confidence information of the vehicle center point is greater than a preset threshold, the position information of the vehicle center point is obtained; based on the global offset information of each vehicle key point and the position information of the vehicle center point, the first position information of each vehicle key point is obtained; according to multiple position points in a preset range corresponding to the first position information of each vehicle key point, the first position information of each vehicle key point is adjusted to obtain the second position information of each vehicle key point; based on the second position information of each vehicle key point and the local offset information, the target position information of each vehicle key point is obtained.

[0009] According to a second aspect of the embodiments of this application, a vehicle key point position detection device is provided, comprising:

[0010] The image acquisition module is used to acquire an image of the target vehicle in front of the vehicle as the image to be recognized.

[0011] The key point related information prediction module is used to input the image to be identified into the key point information extraction model, perform prediction processing on the key point related information of the target vehicle, and obtain the vehicle center point, multiple vehicle key points, and global offset information and local offset information corresponding to each vehicle key point. The global offset information represents the offset of each key point from the vehicle center point, and the local offset information represents the offset between vehicle key points with a preset adjacent relationship.

[0012] The vehicle center point location information acquisition module is used to acquire the vehicle center point location information when the center confidence information of the vehicle center point is greater than a preset threshold.

[0013] The first position information acquisition module is used to obtain the first position information of each vehicle key point based on the global offset information of each vehicle key point and the position information of the vehicle center point.

[0014] The second location information acquisition module is used to adjust the first location information of each vehicle key point according to multiple location points in a preset range corresponding to the first location information of each vehicle key point, so as to obtain the second location information of each vehicle key point.

[0015] The target location information acquisition module is used to obtain the target location information of each vehicle key point based on the second location information and local offset information of each vehicle key point.

[0016] According to a third aspect of the embodiments of this application, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the method as described in any one of the first aspects above.

[0017] According to a fourth aspect of the present application, a computer-readable storage medium is provided, wherein when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform any of the methods described in the first aspect of the present application.

[0018] According to a fifth aspect of the embodiments of this application, a computer program product is provided, including computer instructions that, when executed by a processor, cause a computer to perform the method described in any one of the first aspects of the embodiments of this application.

[0019] The technical solutions provided by the embodiments of this application bring at least the following beneficial effects:

[0020] By acquiring the image to be identified and inputting it into the key point information extraction model, the key point information of the target vehicle is predicted and processed to obtain the vehicle center point, multiple vehicle key points, and global and local offset information corresponding to each vehicle key point. The key point information extraction model can be used to predict the key point information of the target vehicle, enabling end-to-end key point information detection, thereby improving the safety of vehicles in autonomous driving.

[0021] Furthermore, by obtaining the vehicle center point location information when the center confidence information of the vehicle center point is greater than a preset threshold, the reliability of the vehicle center point location information can be improved.

[0022] Furthermore, by using the global offset information of each vehicle key point and the position information of the vehicle center point, the first position information of each vehicle key point is obtained. Based on multiple position points within a preset range corresponding to the first position information of each vehicle key point, the first position information of each vehicle key point is adjusted to obtain the second position information of each vehicle key point. Based on the second position information and local offset information of each vehicle key point, the target position information of each vehicle key point is obtained, which reduces the processing steps of vehicle key points and improves the detection efficiency of vehicle key points. Through local enhancement and global enhancement, the accuracy of vehicle key point position detection is improved.

[0023] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application.

[0024] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0025] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application, and do not constitute an undue limitation of this application.

[0026] Figure 1 This is a schematic diagram illustrating an application environment according to an exemplary embodiment.

[0027] Figure 2 This is a flowchart illustrating a method for detecting the location of key points of a vehicle according to an exemplary embodiment.

[0028] Figure 3 This is a schematic diagram illustrating key points of a vehicle according to an exemplary embodiment.

[0029] Figure 4 This is a flowchart illustrating a key point information extraction model according to an exemplary embodiment.

[0030] Figure 5 This is a block diagram of a device for detecting the location of key points of a vehicle, according to an exemplary embodiment.

[0031] Figure 6 This is a block diagram of an electronic device for object size estimation according to an exemplary embodiment. Figure 1 .

[0032] Figure 7 This is a block diagram of an electronic device for object size estimation according to an exemplary embodiment. Figure 2 . Detailed Implementation

[0033] To enable those skilled in the art to better understand the technical solutions of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0034] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0035] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or computers-controlled machines to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. AI software technology mainly includes computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0036] In recent years, with the research and progress of artificial intelligence technology, artificial intelligence technology has been widely used in many fields. The solutions provided in the embodiments of this application involve technologies such as machine learning / deep learning, which are specifically illustrated through the following embodiments.

[0037] Please see Figure 1 , Figure 1 This is a schematic diagram illustrating an application environment according to an exemplary embodiment, such as... Figure 1 As shown, the application environment may include server 01 and terminal 02.

[0038] In an optional embodiment, server 01 can be used to train an initial machine learning model. Specifically, server 01 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0039] In an optional embodiment, terminal 02 can be used for vehicle key point location detection. Specifically, terminal 02 can be, but is not limited to, electronic devices such as smartphones, desktop computers, tablets, laptops, smart speakers, digital assistants, augmented reality (AR) / virtual reality (VR) devices, and smart wearable devices. Optionally, the operating system running on the electronic device can be, but is not limited to, Android, iOS, Linux, Windows, etc.

[0040] In addition, it should be noted that, Figure 1 The example shown is merely one application environment of the vehicle key point location detection method provided in this application.

[0041] In the embodiments described in this specification, the server 01 and the terminal 02 can be directly or indirectly connected through wired or wireless communication, and this application does not impose any restrictions on this.

[0042] It should be noted that the following diagram illustrates one possible sequence of steps, and it is not strictly required to follow this order. Some steps can be performed in parallel without interdependence. The user information (including but not limited to user device information, user personal information, user behavior information, etc.) and data (including but not limited to data used for display, training data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0043] Please see Figure 2 , Figure 2 This is a flowchart illustrating a method for detecting the location of key points in a vehicle according to an exemplary embodiment, which can be applied to the vehicle control system of a self-driving car, such as... Figure 1 As shown, it may include the following steps.

[0044] In step S201, an image of the target vehicle in front of the vehicle is acquired as the image to be identified.

[0045] The target vehicle can refer to a vehicle in front of the vehicle, such as the car in front. The image to be identified can be captured by the vehicle's front-facing camera or other in-vehicle equipment. The number of target vehicles in the image to be identified can be one or more, and this application does not limit this.

[0046] In this embodiment, the vehicle's front-facing camera captures images of the area in front, including the left front, directly in front, and right front. The captured images are then filtered and identified. If the target vehicle is not found in the filtered images, that image is not considered for identification, and no keypoint prediction processing is performed on it. If the target vehicle is found in the filtered images, those images are used as the images to be identified.

[0047] In step S203, the image to be identified is input into the key point information extraction model, and the key point information of the target vehicle is predicted and processed to obtain the vehicle center point, multiple vehicle key points, and global offset information and local offset information corresponding to each vehicle key point.

[0048] The global offset information represents the offset of each key point from the vehicle center point, while the local offset information represents the offset between vehicle key points with a pre-defined adjacency relationship. The key point information extraction model can refer to an algorithm or model used to extract vehicle key point information from the image to be identified. The vehicle center point can be the mean point of all vehicle key points.

[0049] Vehicle key points can refer to multiple points used to detect the main outline of a vehicle. Figure 3 This is a schematic diagram illustrating key points of a vehicle according to an exemplary embodiment. For example... Figure 3 As shown, 12 vehicle key points can be set: left rear chassis, right rear chassis, right front chassis, left front chassis, left rear roof, right rear roof, right front roof, left front roof, left rear taillight, right rear taillight, right front headlight, and left front headlight. Figure 3 The 12 key vehicle points marked 0-11 correspond to the above-mentioned locations, and this application does not limit them.

[0050] In the embodiments of this specification, the acquired image to be identified is input into the key point information extraction model. Based on the relevant information of the key points of the target vehicle, the key points of the target vehicle in the image to be processed are predicted and analyzed. Then, four branches corresponding to the target vehicle can be obtained, namely the initial position information of multiple vehicle key points, the initial position information of the vehicle center point, the global offset information for global information enhancement, and the local offset information for local enhancement.

[0051] In one possible implementation, the keypoint information extraction model includes five modules: a downsampling module, an attention module, an upsampling and convolution module, a feature fusion module, and a keypoint prediction module. The downsampling module downsamples the image to be recognized, obtaining a downsampled image. This downsampled image is then input into the attention module for weighted processing, resulting in a corrected feature image. The corrected feature image is then input into the upsampling and convolution module to obtain an upsampled image. This upsampled image is then input into the feature fusion module to obtain a fused feature image. Finally, the fused feature image is input into the keypoint prediction module to predict the keypoints of the target vehicle, obtaining the vehicle center point, multiple vehicle keypoints, and the global and local offset information corresponding to each vehicle keypoint.

[0052] The downsampling module can refer to downsampling the image to be recognized. Downsampling can be done by reducing the image size or reducing the image quality, which is not limited in this application. The attention module can refer to weighted processing of the downsampled image.

[0053] The attention module consists of two parts: channel attention and spatial attention. The channel attention part models the correlation between channels, acquiring global information for each channel through global average pooling and generating channel attention vectors using fully connected layers to control the importance of each channel. The spatial attention part focuses on spatial information in the feature map, capturing spatial dimensional information through different operations (such as max pooling and average pooling), and then applying fully connected layers to obtain spatial attention vectors to adjust the spatial distribution of the upsampled image.

[0054] The upsampling and convolution modules can refer to those used to perform upsampling and convolution processing on the corrected feature image. Upsampling can refer to increasing the size or resolution of the corrected feature image, which is not limited in this application. Convolution processing can be used to extract feature information related to key points in the corrected feature image.

[0055] The feature fusion module can refer to the module used to fuse information related to key points in an upsampled image.

[0056] The key point prediction module can refer to the module used to predict the location information of two key points in a feature fusion image.

[0057] In this embodiment, the image to be recognized is input into the downsampling module to perform downsampling processing, for example, reducing the resolution or size of the image to be recognized, thereby reducing the amount of data and computational load, resulting in a processed downsampled image. The downsampled image is then input into the attention module to enhance the focus on different parts of the target vehicle in the downsampled image. Further, the features of the vehicle key points in the downsampled image are weighted and filtered to obtain a corrected feature image. The corrected feature image is then input into the upsampling and convolution module to upsample the corrected feature image, increasing its resolution. Simultaneously, convolution processing is applied to refine and enrich the features of the vehicle key points in the corrected feature image, resulting in an upsampled image. Finally, the processed upsampled image is input into the feature fusion module, which fuses the features of the vehicle key points in the images processed by the downsampling module, attention module, and upsampling and convolution modules, generating a feature fusion module with more information. The feature fusion image is then input into the keypoint prediction module, where vehicle keypoint processing is performed to obtain four branches of information: vehicle center point, vehicle keypoints, global offset information corresponding to the vehicle keypoints, and local offset information. The target location information of the vehicle keypoints is then obtained by combining these four branches.

[0058] In another possible implementation, the downsampling module includes a multi-level downsampling sub-module, and the downsampled image includes downsampled images of various resolutions. The multi-level downsampling sub-module is used to downsample the image to be recognized to obtain downsampled images of various resolutions.

[0059] Accordingly, the attention module includes attention sub-modules corresponding to the multi-level downsampling sub-modules respectively; the downsampled image is input into the attention module, weighted, and a corrected feature image is obtained, including:

[0060] Downsampled images of various resolutions are input into attention submodules corresponding to downsampled images of various resolutions, and weighted processing is performed to obtain multi-level corrected feature images;

[0061] The multi-level upsampling and convolution module includes upsampling and convolution sub-modules corresponding to the multi-level downsampling sub-module, respectively; the corrected feature image is input into the upsampling and convolution module to obtain an upsampled image, including:

[0062] The feature images after multi-level correction are input into the upsampling and convolution sub-modules corresponding to the multi-level downsampling sub-modules to obtain upsampled images of various resolutions.

[0063] Upsampled images at multiple resolutions are input into the feature fusion module to obtain a feature fusion image;

[0064] The feature fusion image is input into the key point prediction module to perform key point prediction processing for the target vehicle, thereby obtaining the vehicle center point, multiple vehicle key points, and global and local offset information corresponding to each vehicle key point.

[0065] The multi-level downsampling submodule can be used to process images of various resolutions. The attention submodule is used to perform weighted processing on the multi-level downsampled images. The upsampling and convolution submodules can be used to upsample and convolve the feature images after multi-level correction.

[0066] In this embodiment, the image to be processed is input into a multi-level downsampling submodule of the downsampling module for downsampling at different resolutions, resulting in downsampled images of various resolutions. These downsampled images are then input into corresponding multi-level attention submodules for weighted processing, yielding multi-level corrected feature images. Further, the multi-level corrected feature images are input into corresponding multi-level upsampling and convolution submodules to obtain upsampled images of various resolutions. These upsampled images are then input into a feature fusion module for feature fusion processing, resulting in a fused feature image. Finally, the fused feature image is input into a keypoint prediction module for keypoint prediction of the target vehicle. This prediction process involves four branches corresponding to the target vehicle: the vehicle center point, multiple vehicle keypoints, and the global and local offset information for each vehicle keypoint. Combining the predicted parameter information from the four branches, the target position information of the vehicle keypoints is finally obtained.

[0067] In one example Figure 4 This is a flowchart illustrating a key point information extraction model according to an exemplary embodiment. Figure 4 For example, the image to be processed, captured by the vehicle, is input into the downsampling module of the key point information extraction model. The downsampling module includes three sub-modules: downsampling sub-module 1, downsampling sub-module 2, and downsampling sub-module 3. Different downsampling factors can be set to downsample the image to be processed, corresponding to the three downsampling sub-modules. After downsampling, the image to be processed yields three corresponding downsampled images: downsampled image 1, downsampled image 2, and downsampled image 3.

[0068] Taking the downsampling submodule 1 as an example, after downsampling the image to be processed, downsampled image 1 is obtained. Downsampled image 1 is then input into the attention submodule 1 of the corresponding attention module, where it undergoes weighted processing to obtain a corrected feature image 1. Further, the corrected feature image 1 is input into the upsampling and convolution submodule 1 of the corresponding upsampling and convolution module for upsampling and convolution processing to obtain upsampled image 1. At this point, upsampled images 1, 2, and 3 obtained from the three submodules are input sequentially or in parallel into the feature fusion module; this application does not limit this process. Further still, the feature fusion module fuses the features from the three upsampled images to obtain a feature fusion image, which is then input into the keypoint prediction module for vehicle keypoint prediction processing.

[0069] In step S205, if the center confidence information of the vehicle center point is greater than a preset threshold, the position information of the vehicle center point is obtained.

[0070] The center confidence information can refer to information used to determine the accuracy of the vehicle's center point. The preset threshold can refer to a pre-set value, such as 0.6, which is not limited in this application. The location information of the vehicle's center point can refer to the specific coordinates of the vehicle's center point in the image, which can be represented by pixel coordinates, such as (x, y), which is not limited in this application.

[0071] In one possible implementation, the initial position information of each vehicle key point is added together and then averaged to obtain the initial position information of the vehicle center point; based on the initial position information of the vehicle center point, the center confidence information corresponding to the vehicle center point is determined.

[0072] The initial position information of the vehicle key points can refer to the information related to the position of the vehicle key points obtained after predicting and processing the relevant information of the target vehicle key points.

[0073] In the embodiments of this specification, the key points of the vehicle in the image are predicted to obtain the initial position information of the key points of the vehicle. The initial position information is added together and then averaged to obtain the initial position information of the vehicle center point. Then, the confidence information of the center point of the target vehicle is determined according to the trained model.

[0074] In one example, 12 key points are set in the target vehicle. A coordinate axis is established with the top left corner of the image to be processed as the origin, and the coordinate positions of the 12 key points are determined, i.e., the initial position information of the key points. The x-coordinates of the 12 key points are added together and then divided by 12 to obtain the x-coordinate of the vehicle's center point. The y-coordinate of the vehicle's center point is then obtained using the same method. Based on the x-coordinates and y-coordinates, the initial position information of the vehicle's center point is finally obtained. Based on the initial position information of the vehicle's center point, the confidence information of the center point is determined.

[0075] In another possible implementation, if the confidence level of the vehicle center point is less than or equal to a preset threshold, an image of the target vehicle in front of the vehicle is reacquired as the image to be identified, and the prediction processing step is returned.

[0076] In the embodiments of this specification, after obtaining the vehicle confidence score of the target vehicle, if the vehicle center point confidence score is less than or equal to a preset threshold, it may be because the captured image to be processed is too blurry. This application does not limit this. The image to be identified is reacquired, and prediction processing is performed on the image to be identified to obtain a new vehicle center point confidence score.

[0077] In step S207, the first position information of each vehicle key point is obtained based on the global offset information of each vehicle key point and the position information of the vehicle center point.

[0078] In the embodiments of this specification, key point-related information in the image to be processed is predicted to obtain global offset information, which is the deviation between the initial position information of each vehicle key point and the initial position information of the vehicle center point. The global offset information of each vehicle key point is added to the position information of the vehicle center point; for example, the abscissa of the global offset information is added to the abscissa of the vehicle center point to obtain the abscissa of the first position information of the vehicle key point. Then, using the same method, the ordinate of the first position information of the vehicle key point is obtained. Based on the abscissa and ordinate of the first position information of the vehicle key point, the first position information of the vehicle key point can be obtained.

[0079] In step S209, the first position information of each vehicle key point is adjusted according to multiple position points in a preset range corresponding to the first position information of each vehicle key point to obtain the second position information of each vehicle key point.

[0080] In one possible implementation, a preset range is obtained with each vehicle key point as the center and a preset distance as the radius, and multiple location points are extracted from the preset range; the location confidence information corresponding to each of the multiple location points is determined; the target location point corresponding to the highest location confidence information is selected from the multiple location points; and the location information of the target location point is determined as the second location information of the corresponding vehicle key point.

[0081] In this embodiment, using the first location information of each vehicle key point as the center and a preset distance as the radius (the preset distance can be 5 pixels, which is not limited in this application), a preset range for each vehicle key point can be obtained. Based on the trained model data, the location confidence information of each vehicle key point is determined, and the confidence information of multiple location points is extracted from the preset range. The location confidence information of the vehicle key points and the confidence information of the multiple location points extracted from the preset range are sorted by confidence, and the target location point with the highest location confidence information is selected from the sorted list. The location information of each target location point is determined as the second location information of the corresponding vehicle key point.

[0082] In step S211, the target position information of each vehicle key point is obtained based on the second position information and local offset information of each vehicle key point.

[0083] In one example, taking Table 1 below as an example, the local information of each key point is enhanced by using adjacent key point pairs. Table 1 includes 12 vehicle key points from 0 to 11 and 3 adjacent key points. This application does not limit the number of adjacent key points.

[0084] Taking the left rear chassis (0) as an example among the vehicle key points, the corresponding adjacent key points are the right rear chassis, the left front chassis, and the left rear taillight. Connecting the left rear chassis (0) with the corresponding adjacent key points yields the distance and angle information between the vehicle key points and the adjacent key points. Based on the trained angle and distance information, the local offset information of (0) and other adjacent key points can be obtained. Further, based on the other vehicle key points, corresponding processing is performed to obtain the local offset information corresponding to each vehicle key point. The model has been trained on the vehicle's angle and connection relationship. That is, when the right rear chassis of the adjacent key point is occluded, the position information of the right rear chassis can be obtained based on the connection relationship between the left rear chassis (0), the left front chassis, and the left rear taillight. According to the above method, the position information of all vehicle key points can be obtained. Then, by adding the second position information of each vehicle key point with the corresponding local offset information, the final target position information of the vehicle key points is obtained.

[0085] Table 1

[0086] Vehicle key points Adjacent key points Left rear chassis (0) Right rear chassis, left front chassis, left rear taillight Right rear chassis (1) Left rear chassis, right front chassis, right rear taillight Right front chassis (2) Left front chassis, right front chassis, right front headlight Left front chassis (3) Right front chassis, left rear chassis, left front headlight Left rear roof (4) Right rear roof, left front roof, left rear taillight Right rear roof (5) Left rear roof, right front roof, right rear taillight Right front roof (6) Right rear roof, left front roof, right front headlight Left front roof (7) Right front roof, left rear roof, left front headlight Left rear taillight (8) Right rear taillight, left front taillight, left rear roof Right rear taillight (9) Left rear taillight, right front taillight, right rear roof Right front headlight (10) Right rear taillight, left front headlight, right front roof Left front headlight (11) Left rear taillight, left front roof, right front headlight

[0087] In one possible implementation, the above method may further include a training process for a key point information extraction model. Specifically, it may include: acquiring multiple sample images and key point location label information corresponding to each sample image; the sample images include sample vehicles; inputting the sample images into an initial machine learning model to perform prediction processing on the key point related information of the sample vehicles, obtaining the predicted center point corresponding to the sample vehicles, multiple predicted key points, and global and local prediction offset information corresponding to each predicted key point; if the center confidence information of the predicted center point is greater than a preset threshold, acquiring the location information of the predicted center point; and based on the global offset corresponding to each predicted key point... The third position information of each predicted key point is obtained from the quantity information and the position information of the predicted center point. Based on multiple position points within a preset range corresponding to the third position information of each predicted key point, the third position information of each predicted key point is adjusted to obtain the fourth position information of each predicted key point. Based on the fourth position information of each predicted key point and the local offset information, the predicted position information of each vehicle key point is obtained. Based on the predicted position information and the key point position label information, the position loss information is determined. Based on the position loss information, the parameters of the initial machine learning model are adjusted until the position loss information meets the loss threshold. The initial machine learning model that meets the loss threshold is determined as the key point information extraction model.

[0088] The sample images can be used to train an initial machine learning model; for example, a large number of images of vehicles traveling on the road can be obtained as sample images. Keypoint location label information refers to the pre-annotated keypoint location information within the sample vehicles in the sample images. The initial machine learning model can be a pre-defined neural network, such as a convolutional neural network; this application does not limit its scope.

[0089] The prediction center point can be the mean point of all prediction key points. Prediction key points can be multiple points used to detect the main outline of the predicted vehicle. Global prediction offset information represents the offset of each prediction key point from the predicted center point of the sample vehicle, while local prediction offset information represents the offset between prediction key points of the sample vehicle with a pre-defined adjacency relationship. The position loss function is used to measure the loss between the model's predicted position and the actual position.

[0090] In the embodiments of this specification, the specific implementation process of predicting key point-related information during training, obtaining the location information of the predicted center point, the third location information, the fourth location information, and the predicted location information can be found in steps S203 to S211 above, and will not be repeated here. Accordingly, after obtaining the predicted location information of each vehicle's key points, the predicted location information of each sample vehicle's key points is compared with the corresponding key point location label information, and location loss information is determined. When the loss information does not meet the loss threshold, the parameters of the initial machine learning are adjusted. For example, the gradient can be calculated based on the location loss information, and gradient backpropagation is used to adjust the parameters of the initial machine learning. The calculation of the location loss information can be based on a preset loss function; this application does not limit the preset loss function. Furthermore, the initial machine learning model that meets the loss function requirements can be determined as the key information extraction model.

[0091] Figure 5 This is a block diagram illustrating a vehicle key point position detection device according to an exemplary embodiment. (Refer to...) Figure 5 The device may include:

[0092] The image acquisition module 501 is used to acquire an image of the target vehicle in front of the vehicle as the image to be recognized.

[0093] The key point related information prediction module 503 is used to input the image to be identified into the key point information extraction model, perform prediction processing on the key point related information of the target vehicle, and obtain the vehicle center point, multiple vehicle key points, and global offset information and local offset information corresponding to each vehicle key point. The global offset information represents the offset of each key point from the vehicle center point, and the local offset information represents the offset between vehicle key points with a preset adjacent relationship.

[0094] In one possible implementation, the key-point related information prediction module 503 includes:

[0095] The vehicle center point acquisition module is used to add and average the initial position information of each vehicle key point to obtain the initial position information of the vehicle center point.

[0096] The center confidence information confirmation module is used to determine the center confidence information corresponding to the vehicle's center point based on the initial position information of the vehicle's center point.

[0097] In one possible implementation, the key-point related information prediction module 503 further includes:

[0098] The downsampling module is used to downsample the image to be recognized to obtain a downsampled image.

[0099] The module for obtaining the corrected feature image is used to input the downsampled image into the attention module for weighted processing to obtain the corrected feature image.

[0100] The upsampled image acquisition module is used to input the corrected feature image into the upsampling and convolution module to obtain the upsampled image;

[0101] The feature fusion image acquisition module is used to input the upsampled image into the feature fusion module to obtain the feature fusion image;

[0102] The prediction processing module is used to input the feature fusion image into the key point prediction module to perform key point prediction processing for the target vehicle, and obtain the vehicle center point, multiple vehicle key points, and global offset information and local offset information corresponding to each vehicle key point.

[0103] The vehicle center point location information acquisition module 505 is used to acquire the vehicle center point location information when the center confidence information of the vehicle center point is greater than a preset threshold.

[0104] In one possible implementation, the vehicle center point location information acquisition module 505 includes:

[0105] If the confidence level of the vehicle center point is less than or equal to a preset threshold, the image of the target vehicle in front of the vehicle is reacquired as the image to be identified, and the prediction processing step is returned.

[0106] The first position information acquisition module 507 is used to obtain the first position information of each vehicle key point based on the global offset information of each vehicle key point and the position information of the vehicle center point.

[0107] The second location information acquisition module 509 is used to adjust the first location information of each vehicle key point according to multiple location points in a preset range corresponding to the first location information of each vehicle key point, so as to obtain the second location information of each vehicle key point.

[0108] In one possible implementation, the second location information acquisition module 509 includes:

[0109] The location point extraction module is used to obtain the preset range with each vehicle key point as the center and a preset distance as the radius, and to extract the multiple location points from the preset range;

[0110] The location confidence information determination module is used to determine the location confidence information corresponding to each of the plurality of location points;

[0111] The filtering module is used to filter out the target location point with the highest location confidence information from the plurality of location points;

[0112] The second location information determination module is used to determine the location information of the target location point as the second location information of the corresponding vehicle key point.

[0113] The target location information acquisition module 511 is used to obtain the target location information of each vehicle key point based on the second location information and local offset information of each vehicle key point.

[0114] In one possible implementation, a vehicle key point position detection device further includes:

[0115] The key point location label information acquisition module is used to acquire multiple sample images and the key point location label information corresponding to each sample image; the sample images include sample vehicles.

[0116] The prediction information acquisition module is used to input the sample image into the initial machine learning model, perform prediction processing on the key point related information of the sample vehicle, and obtain the prediction center point, multiple prediction key points, and global prediction offset information and local prediction offset information corresponding to each prediction key point of the sample vehicle.

[0117] The prediction center point acquisition module is used to acquire the location information of the prediction center point when the center confidence information of the prediction center point is greater than a preset threshold.

[0118] The third position information determination module is used to obtain the third position information of each predicted key point based on the global offset information corresponding to each predicted key point and the position information of the predicted center point.

[0119] The fourth location information determination module is used to adjust the third location information of each predicted key point based on multiple location points in a preset range corresponding to the third location information of each predicted key point, so as to obtain the fourth location information of each predicted key point.

[0120] The predicted location information determination module is used to obtain the predicted location information of each vehicle key point based on the fourth location information and local offset information of each predicted key point.

[0121] The location loss information determination module is used to determine the location loss information based on the predicted location information and the key point location label information.

[0122] The key point information extraction model determination module is used to adjust the parameters of the initial machine learning model based on the location loss information until the location loss information meets the loss threshold. The initial machine learning model that meets the loss threshold is then determined as the key point information extraction model.

[0123] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0124] Figure 6 This is a block diagram illustrating an electronic device for detecting the location of key points on a vehicle, according to an exemplary embodiment. The electronic device may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown, the electronic device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a method for detecting the location of key points on a vehicle. The display screen can be a liquid crystal display (LCD) or an e-ink display. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the device's casing, or an external keyboard, touchpad, or mouse.

[0125] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0126] Figure 7 This is a block diagram illustrating an electronic device for detecting the location of key points on a vehicle, according to an exemplary embodiment. The electronic device may be a server, and its internal structure diagram may be as follows: Figure 7 As shown, the electronic device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a method for detecting the location of key points on a vehicle.

[0127] Those skilled in the art will understand that Figure 7The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0128] In an exemplary embodiment, an electronic device is also provided, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement a vehicle key point position detection method as described in the embodiments of this application.

[0129] In an exemplary embodiment, a computer-readable storage medium is also provided, which, when executed by a processor of an electronic device, enables the electronic device to perform a vehicle key point detection method according to an embodiment of this application. The computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, or optical data storage device, etc.

[0130] In an exemplary embodiment, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute a vehicle key point location detection method according to an embodiment of this application.

[0131] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0132] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0133] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for detecting the location of key points on a vehicle, characterized in that, include: Acquire an image of the target vehicle in front of the vehicle as the image to be identified; The image to be identified is input into the key point information extraction model, and the downsampling module is used to downsample the image to be identified to obtain a downsampled image. The downsampled image is input into the attention module for weighted processing to obtain a corrected feature image. The corrected feature image is then input into the upsampling and convolution module to obtain an upsampled image. The upsampled image is then input into the feature fusion module to obtain a fused feature image. The fused feature image is then input into the keypoint prediction module for keypoint prediction of the target vehicle, resulting in the vehicle center point, multiple vehicle keypoints, and global and local offset information for each vehicle keypoint. The global offset information represents the offset between each keypoint and the vehicle center point, and the local offset information represents the offset between vehicle keypoints with a preset adjacent relationship. The keypoint information extraction model includes the downsampling module, the attention module, the upsampling and convolution module, the feature fusion module, and the keypoint prediction module. If the center confidence information of the vehicle center point is greater than a preset threshold, the position information of the vehicle center point is obtained. Based on the global offset information of each vehicle key point and the position information of the vehicle center point, the first position information of each vehicle key point is obtained. Based on multiple location points within a preset range corresponding to the first location information of each vehicle key point, adjust the first location information of each vehicle key point to obtain the second location information of each vehicle key point. Based on the second position information of each vehicle key point and the local offset information, the target position information of each vehicle key point is obtained.

2. The method according to claim 1, characterized in that, The downsampling module includes multi-level downsampling sub-modules, and the downsampling image includes downsampling images of various resolutions; the step of using the downsampling module to downsample the image to be identified to obtain a downsampling image includes: The multi-level downsampling submodule is used to downsample the image to be identified to obtain downsampled images of various resolutions; Accordingly, the attention module includes attention sub-modules corresponding to the multi-level downsampling sub-modules respectively; the step of inputting the downsampled image into the attention module, performing weighted processing, and obtaining the corrected feature image includes: The downsampled images of various resolutions are input into the attention submodule corresponding to the downsampled images of various resolutions, and weighted processing is performed to obtain the feature images after multi-level correction. The upsampling and convolution module includes upsampling and convolution sub-modules corresponding to the multi-level downsampling sub-modules, respectively; the step of inputting the corrected feature image into the upsampling and convolution module to obtain the upsampled image includes: The multi-level corrected feature images are input into the upsampling and convolution sub-modules corresponding to the multi-level downsampling sub-modules respectively to obtain upsampled images of various resolutions; The upsampled images at various resolutions are input into the feature fusion module to obtain a feature fusion image; The feature fusion image is input into the key point prediction module to perform key point prediction processing for the target vehicle, thereby obtaining the vehicle center point, multiple vehicle key points, and global and local offset information corresponding to each vehicle key point.

3. The method according to claim 1, characterized in that, Based on multiple location points within a preset range corresponding to the first location information of each vehicle's key points, the first location information of each vehicle's key points is adjusted to obtain the second location information of each vehicle's key points, including: Using each vehicle's key point as the center and a preset distance as the radius, the preset range is obtained, and the multiple location points are extracted from the preset range; Determine the location confidence information corresponding to each of the plurality of location points; Filter out the target location point with the highest location confidence information from the plurality of location points; The location information of the target location point is determined as the second location information of the corresponding vehicle key point.

4. The method according to claim 1, characterized in that, The method further includes: The initial position information of each vehicle's key point is summed and then averaged to obtain the initial position information of the vehicle's center point. Based on the initial position information of the vehicle center point, determine the center confidence information corresponding to the vehicle center point.

5. The method according to claim 1, characterized in that, The method further includes: If the center confidence information of the vehicle center point is less than or equal to the preset threshold, the image of the target vehicle in front of the vehicle is reacquired as the image to be identified, and the process returns to the prediction step.

6. The method according to claim 1, characterized in that, The method further includes: Acquire multiple sample images and key point location label information corresponding to each sample image; the sample images include sample vehicles; The sample image is input into the initial machine learning model to perform prediction processing on the key point information of the sample vehicle, thereby obtaining the prediction center point, multiple prediction key points, and global prediction offset information and local prediction offset information corresponding to each prediction key point of the sample vehicle. If the center confidence information of the predicted center point is greater than the preset threshold, the location information of the predicted center point is obtained. Based on the global prediction offset information corresponding to each prediction key point and the position information of the prediction center point, the third position information of each prediction key point is obtained. Based on multiple location points within a preset range corresponding to the third location information of each predicted key point, adjust the third location information of each predicted key point to obtain the fourth location information of each predicted key point. Based on the fourth position information of each predicted key point and the local predicted offset information, the predicted position information of each predicted key point is obtained. Based on the predicted location information and the key point location label information, determine the location loss information; The parameters of the initial machine learning model are adjusted based on the location loss information until the location loss information meets the loss threshold. The initial machine learning model that meets the loss threshold is then determined as the key point information extraction model.

7. A vehicle key point detection device, characterized in that, include: The image acquisition module is used to acquire an image of the target vehicle in front of the vehicle as the image to be recognized. The key point related information prediction module is used to input the image to be identified into the key point information extraction model, and use the downsampling module to downsample the image to be identified to obtain a downsampled image. The downsampled image is input into the attention module for weighted processing to obtain a corrected feature image. The corrected feature image is then input into the upsampling and convolution module to obtain an upsampled image. The upsampled image is then input into the feature fusion module to obtain a fused feature image. The fused feature image is then input into the keypoint prediction module for keypoint prediction of the target vehicle, resulting in the vehicle center point, multiple vehicle keypoints, and global and local offset information for each vehicle keypoint. The global offset information represents the offset between each keypoint and the vehicle center point, and the local offset information represents the offset between vehicle keypoints with a preset adjacent relationship. The keypoint information extraction model includes the downsampling module, the attention module, the upsampling and convolution module, the feature fusion module, and the keypoint prediction module. The vehicle center point location information acquisition module is used to acquire the location information of the vehicle center point when the center confidence information of the vehicle center point is greater than a preset threshold. The first location information acquisition module is used to obtain the first location information of each vehicle key point based on the global offset information of each vehicle key point and the location information of the vehicle center point. The second location information acquisition module is used to adjust the first location information of each vehicle key point according to multiple location points in a preset range corresponding to the first location information of each vehicle key point, so as to obtain the second location information of each vehicle key point. The target location information acquisition module is used to obtain the target location information of each vehicle key point based on the second location information of each vehicle key point and the local offset information.

8. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the vehicle key point location detection method as described in any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is enabled to perform the vehicle key point location detection method as described in any one of claims 1 to 5.

Images