Vehicle trunk opening method and device, electronic equipment, medium and vehicle

Abstract

The invention relates to a vehicle trunk opening method and device, electronic equipment, a medium and a vehicle, in particular to the technical field of vehicles. The method comprises the following steps: acquiring an image sequence; inputting the image sequence into a human body key point detection model to obtain a human body key point coordinate sequence output by the human body key point detection model; inputting the human body key point coordinate sequence into an action recognition model to obtain a predicted action output by the action recognition model; when the predicted action is matched with the preset action, judging whether a foot key point coordinate sequence in the human body key point coordinate sequence is in a preset area or not; and under the condition that the foot key point coordinate sequence is located in the preset area, the vehicle trunk is controlled to be opened. The trunk can be opened across the air without operating an entity switch, and the convenience is high; an image recognition technology which is slightly influenced by the environment is utilized, so that the stability is higher; and the action interference of other irrelevant personnel far away from the vehicle can be eliminated, so that the opening of the trunk of the vehicle can be accurately controlled.

Description

Technical Field

[0001] The present disclosure relates to the field of vehicle technology, and in particular to a method, device, electronic equipment, medium and vehicle for opening a vehicle trunk. Background Art

[0002] Currently, the ways to open the trunk of a vehicle include manual switch, voice control, kicking under the rear bumper, and image recognition. Manual switch is very inconvenient when the user is holding an object. Voice control is easily disturbed by noise and has poor stability.

[0003] The "kick" method under the rear bumper is to install a capacitive sensor under the rear bumper. The user makes a specified action with his foot, and the capacitive sensor converts the user's action signal into a high-low level signal, and then judges the high-low level signal to determine whether the user has made a preset action. After the verification, the trunk is opened, but the user's operating range under this scheme is limited to the rear bumper position. Therefore, a convenient, efficient and safe method for opening the vehicle trunk is urgently needed. Summary of the invention

[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a method, device, electronic device, medium and vehicle for opening a vehicle trunk, which can conveniently, efficiently and safely control the opening of the vehicle trunk.

[0005] In order to achieve the above objectives, the technical solutions provided by the embodiments of the present disclosure are as follows:

[0006] In a first aspect, the present disclosure provides a method for opening a vehicle trunk, comprising:

[0007] Acquire an image sequence, wherein the image sequence includes multiple frames of images acquired by a target vehicle-mounted camera at different times;

[0008] The image sequence is input into the human key point detection model to obtain the human key point coordinate sequence output by the human key point detection model; the human key point coordinate sequence is input into the action recognition model to obtain the predicted action output by the action recognition model;

[0009] When the predicted action matches the preset action, determining whether the foot key point coordinate sequence in the human body key point coordinate sequence is in the preset area;

[0010] When the coordinate sequence of the key points of the foot is in the preset area, the vehicle trunk is controlled to open.

[0011] As an optional implementation of the embodiment of the present disclosure, the human key point detection model includes a human detection model and a key point detection model connected in sequence, and an image sequence is input into the human key point detection model to obtain a human key point coordinate sequence output by the human key point detection model, including:

[0012] Input the image sequence into the human body detection model to obtain the coordinates of the human body external frame output by the human body detection model;

[0013] The image sequence is cropped according to the coordinates of the human body external frame to obtain a portrait sequence;

[0014] The portrait sequence is input into the key point detection model to obtain the human body key point coordinate sequence output by the key point detection model.

[0015] As an optional implementation of the embodiment of the present disclosure, a method for training a human body detection model includes:

[0016] Obtain a sample image and a first annotated image, where the first annotated image includes coordinates of a human body bounding box corresponding to the sample image;

[0017] Inputting the sample image and the first annotated image into the first pre-trained model to obtain the predicted bounding box coordinates output by the first pre-trained model;

[0018] The first pre-trained model is trained according to the predicted bounding box coordinates and the human body bounding box coordinates corresponding to the sample image to obtain a converged human body detection model.

[0019] As an optional implementation of the embodiment of the present disclosure, a training method for a key point detection model includes:

[0020] Acquire a sample image and a second annotated image, wherein the second annotated image includes coordinates of key points corresponding to a human body region in the sample image;

[0021] Input the sample image and the second annotated image into the second pre-trained model to obtain the predicted key point coordinates output by the second pre-trained model;

[0022] The second pre-trained model is trained according to the predicted key point coordinates and the key point coordinates corresponding to the human body area in the sample image to obtain a converged key point detection model.

[0023] As an optional implementation of the embodiment of the present disclosure, a method for training an action recognition model includes:

[0024] Obtaining a sample key point coordinate sequence and a labeling action corresponding to the sample key point coordinate sequence;

[0025] Inputting the key point coordinate sequence and the annotation action into a third pre-training model, wherein the third pre-training model includes a feature extraction module, a feature fusion module and a classification module;

[0026] According to the input action of the third pre-trained model and the labeled action corresponding to the sample key point coordinate sequence, the third pre-trained model is trained to obtain a converged action recognition model.

[0027] As an optional implementation of the embodiment of the present disclosure, inputting a human body key point coordinate sequence into an action recognition model to obtain a predicted action output by the action recognition model includes:

[0028] Extract the leg key point coordinate sequence from the human body key points;

[0029] The leg key point coordinate sequence is input into the action recognition model to obtain the predicted action output by the action recognition model.

[0030] As an optional implementation of the embodiment of the present disclosure, when the predicted action matches the preset action, determining whether the foot key point coordinate sequence in the human body key point coordinate sequence is in a preset area includes:

[0031] When the predicted action matches the preset action, the foot key point coordinate sequence in the image coordinate system is converted into a target coordinate sequence in the global coordinate system according to the internal and external parameters of the fisheye camera, and the target coordinate sequence includes the foot key point coordinates in the global coordinate system;

[0032] In the global coordinate system, determine whether the target coordinate sequence is in the preset area.

[0033] In a second aspect, the present disclosure provides a vehicle trunk opening device, the device comprising:

[0034] An acquisition module, used to acquire an image sequence, wherein the image sequence includes multiple frames of images acquired by a target vehicle-mounted camera at different times;

[0035] The key point detection module is used to input the image sequence into the human key point detection model to obtain the human key point coordinate sequence output by the human key point detection model; the action recognition module is used to input the human key point coordinate sequence into the action recognition model to obtain the predicted action output by the action recognition model;

[0036] A judgment module, used for judging whether the foot key point in the human body key point coordinate sequence is in a preset area when the predicted action matches the preset action;

[0037] The control module is used to control the opening of the vehicle trunk when the key point of the foot is in a preset area.

[0038] As an optional implementation of the embodiment of the present disclosure, the human key point detection model includes a human detection model and a key point detection model connected in sequence, and the key point detection module is specifically used to: input the image sequence into the human detection model to obtain the coordinates of the human body external frame output by the human detection model;

[0039] The image sequence is cropped according to the coordinates of the human body external frame to obtain a portrait sequence;

[0040] The portrait sequence is input into the key point detection model to obtain the human body key point coordinate sequence output by the key point detection model.

[0041] As an optional implementation of the embodiment of the present disclosure, the device further includes: a model training module, which is used to train the human body detection model included in the human body key point detection model, specifically used to:

[0042] Obtain a sample image and a first annotated image, where the first annotated image includes coordinates of a human body bounding box corresponding to the sample image;

[0043] Inputting the sample image and the first annotated image into the first pre-trained model to obtain the predicted bounding box coordinates output by the first pre-trained model;

[0044] The first pre-trained model is trained according to the predicted bounding box coordinates and the human body bounding box coordinates corresponding to the sample image to obtain a converged human body detection model.

[0045] As an optional implementation of the embodiment of the present disclosure, the device further includes: a model training module, which is used to train the key point detection model included in the human key point detection model, specifically used to:

[0046] Acquire a sample image and a second annotated image, wherein the second annotated image includes coordinates of key points corresponding to a human body region in the sample image;

[0047] Input the sample image and the second annotated image into the second pre-trained model to obtain the predicted key point coordinates output by the second pre-trained model;

[0048] The second pre-trained model is trained according to the predicted key point coordinates and the key point coordinates corresponding to the human body area in the sample image to obtain a converged key point detection model.

[0049] As an optional implementation of the embodiment of the present disclosure, the device further includes: a model training module, which is used to train the action recognition model, specifically used to:

[0050] Obtaining a sample key point coordinate sequence and a labeling action corresponding to the sample key point coordinate sequence;

[0051] Inputting the key point coordinate sequence and the labeled action into the third pre-trained model;

[0052] The third pre-trained model is trained according to the output actions of the third pre-trained model and the labeled actions corresponding to the sample key point coordinate sequence to obtain a converged action recognition model.

[0053] As an optional implementation of the embodiment of the present disclosure, the action recognition module is specifically used to: extract a leg key point coordinate sequence from a human body key point coordinate sequence;

[0054] The leg key point coordinate sequence is input into the action recognition model to obtain the predicted action output by the action recognition model.

[0055] As an optional implementation of the embodiment of the present disclosure, the judgment module is specifically used to: when the predicted action matches the preset action, convert the foot key point coordinate sequence in the image coordinate system into a target coordinate sequence in the global coordinate system according to the internal and external parameters of the fisheye camera, wherein the target coordinate sequence includes the foot key point coordinates in the global coordinate system;

[0056] In the global coordinate system, determine whether the target coordinate sequence is in the preset area.

[0057] In a third aspect, the present disclosure provides an electronic device comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the method for opening a vehicle trunk as described in the first aspect or any one of its optional embodiments is implemented.

[0058] In a fourth aspect, the present disclosure provides a computer-readable storage medium, comprising: a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the method for opening a vehicle trunk as described in the first aspect or any optional embodiment thereof is implemented.

[0059] In a fifth aspect, an embodiment of the present disclosure provides a vehicle, comprising: a rear-view fisheye camera; and an opening device for a vehicle trunk as described in the second aspect or any optional embodiment thereof, or an electronic device as described in the third aspect.

[0060] In a sixth aspect, the present disclosure provides a computer program product, comprising: the computer program product comprises a computer program, and when the computer program runs on a computer, the computer implements the vehicle trunk opening method as described in the first aspect or any optional embodiment thereof.

[0061] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has the following advantages:

[0062] The disclosed embodiment provides a method for opening a vehicle trunk. The method first acquires an image sequence, which includes multiple frames of images captured by a target vehicle-mounted camera at different times; then the image sequence is input into a human key point detection model to obtain a human key point coordinate sequence output by the human key point detection model; and then the human key point coordinate sequence is input into a motion recognition model to obtain a predicted action output by the motion recognition model. When the predicted action matches a preset action, it is determined whether a foot key point coordinate sequence in the human key point coordinate sequence is in a preset area. If so, it indicates that a user in the preset area has made a preset action to open the trunk, and the vehicle trunk is controlled to open.

[0063] Compared with the manual switch starting method, the present invention does not require the operation of the physical switch, and can realize the remote opening of the trunk, which is very convenient; compared with the voice control starting method, the present invention utilizes image recognition technology that is less affected by the environment and is more stable. In addition, in order to reduce misjudgment, the present invention focuses on the detected coordinate sequence of the key points of the feet, so as to further determine whether the user is in the preset area when the user makes a preset action, thereby eliminating the interference of other irrelevant people at a farther distance from the vehicle, thereby realizing precise control of the opening of the vehicle trunk. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0065] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0066] Figure 1 A schematic flow chart of a method for opening a vehicle trunk according to an embodiment of the present disclosure;

[0067] Figure 2 A schematic diagram of a rear-view fisheye camera provided in an embodiment of the present disclosure;

[0068] Figure 3 A schematic diagram of an application of a human body detection model provided in an embodiment of the present disclosure;

[0069] Figure 4 A schematic diagram of an application of a key point detection model provided in an embodiment of the present disclosure;

[0070] Figure 5 A schematic diagram of an application of the action recognition model provided by an embodiment of the present disclosure;

[0071] Figure 6 A schematic diagram of a preset area provided in an embodiment of the present disclosure;

[0072] Figure 7 It is a structural schematic diagram of a vehicle trunk opening device according to an embodiment of the present disclosure;

[0073] Figure 8 The present invention is a schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0074] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0075] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0076] In order to solve some or all of the technical problems existing in the related art, a method for opening a vehicle trunk provided in the embodiment of the present disclosure can be implemented by an opening device or an electronic device of the vehicle trunk, and the electronic device includes but is not limited to a server, a personal computer, a laptop computer, a tablet computer, a smart phone, a vehicle-mounted device, etc. The operating system of the electronic device may include Android, a mobile operating system (iOS) developed by Apple, an operating system (Windows) developed by Microsoft Corporation of the United States, etc., and the embodiment of the present disclosure does not limit this. The electronic device can be operated alone to implement the present disclosure, or it can be connected to a network and implement the present disclosure through interactive operations with other computer devices in the network. Among them, the network in which the electronic device is located includes but is not limited to the Internet, a wide area network, a metropolitan area network, a local area network, a virtual private network (VPN) network, etc.

[0077] It should be noted that the protection scope of the vehicle trunk opening method described in the embodiment of the present disclosure is not limited to the execution order of the steps listed in this embodiment, and all solutions implemented by adding, reducing or replacing steps in the prior art based on the principles of the present disclosure are included in the protection scope of the present disclosure.

[0078] like Figure 1 As shown, Figure 1 The figure is a flow chart of a method for opening a vehicle trunk according to an embodiment of the present disclosure. The method can be executed by a vehicle trunk opening device, wherein the device can be implemented by software and / or hardware and can generally be integrated in an electronic device. Figure 1 As shown, the method mainly includes the following steps S101 to S105:

[0079] S101: Acquire an image sequence.

[0080] The image sequence includes multiple frames of images captured by the target vehicle-mounted camera at different times, and the image sequence sorts the images according to the acquisition time.

[0081] In the related art, face recognition technology is also used to identify the user approaching the trunk of the vehicle, so that the authorized user can control the opening of the trunk. However, excessive use of face recognition technology can easily lead to information leakage and pose an information security risk. Therefore, in the embodiment of the present disclosure, the target vehicle-mounted camera can be installed in a preset position, which can ensure that each image collected does not contain facial information, that is, each frame of the image in the image sequence only contains other information besides the facial information.

[0082] The target vehicle camera in the embodiment of the present disclosure may be a rear-view fisheye camera. A rear-view fisheye camera is a fisheye camera used to capture images of the rear of a vehicle, and the shooting range of the rear-view fisheye camera includes the trunk area of ​​the vehicle. For example, Figure 2 As shown, Figure 2 This is a schematic diagram of a rear-view fisheye camera provided in an embodiment of the present disclosure. The rear-view fisheye camera 201 is arranged at the position shown in the figure. The present disclosure does not limit the installation position and specific number of the rear-view fisheye camera.

[0083] In some embodiments, the acquisition frequency of the rear-view fisheye camera is set, for example, 10 frames per second, and the acquired image sequence includes multiple frames of images. For example, an image sequence acquired by the rear-view fisheye camera within two seconds is obtained, and the image sequence includes 20 frames of images.

[0084] The disclosed embodiment uses a rear-view fisheye camera to capture images covering the trunk area. Based on the surround-view fisheye camera equipped on most existing vehicles, there is no need to additionally set a capacitive sensor on the rear bumper. It can capture images and based on subsequent visual algorithms, it can effectively improve the control accuracy of the trunk opening and reduce misjudgments.

[0085] S102: Input the image sequence into a human key point detection model to obtain a human key point coordinate sequence output by the human key point detection model.

[0086] Among them, the human body key point detection model includes a human body detection model and a key point detection model connected in sequence.

[0087] The disclosed embodiments provide a method for training a human key point detection model. The following will introduce the training process of the human body detection model and the training process of the key point detection model respectively.

[0088] (1) Human body detection model

[0089] In some embodiments, the training method of the human body detection model includes: obtaining a sample image and a first annotated image, the first annotated image including the coordinates of the human body external frame corresponding to the sample image; inputting the sample image and the first annotated image into a first pre-trained model to obtain the predicted external frame coordinates output by the first pre-trained model; training the first pre-trained model according to the predicted external frame coordinates and the human body external frame coordinates corresponding to the sample image to obtain a converged human body detection model.

[0090] The specific steps include:

[0091] ① Obtain model training set

[0092] The sample image and the first annotated image are obtained as a model training set. The first annotated image includes the coordinates of the human body external bounding box corresponding to the sample image. It can be understood that the first annotated image annotates the human body position based on the sample image and annotates the human body external bounding box corresponding to the human body. Optionally, the sample image includes positive sample images and negative sample images. The positive sample image is an image containing a human body, and the negative sample image is an image not containing a human body. Combining positive and negative sample images for model training can improve the robustness and accuracy of the model.

[0093] ②Build the initial model

[0094] The first pre-trained model as the initial model includes a backbone network, a shoulder network, and a head network.

[0095] ③Model training

[0096] Input the sample image and the first annotated image into the first pre-trained model, obtain the predicted human body external bounding box coordinates output by the first pre-trained model, compare the predicted human body external bounding box coordinates with the real human body external bounding box coordinates (the human body external bounding box coordinates corresponding to the sample image), and calculate the corresponding loss function values ​​of the two based on the first preset loss function. When the loss function value is greater than the preset threshold, adjust the model parameters of the first pre-trained model, and then perform a new round of training until the loss function value is less than or equal to the preset loss function value, and obtain a converged human body detection model. Among them, the first preset loss function includes but is not limited to the cross entropy loss function. The preset loss function value is the critical value of the loss function that characterizes the convergence of the model. If it is greater than the preset loss function, it means that the model has not converged, and if it is less than or equal to the preset loss function value, it means that the model has converged.

[0097] (2) Key point detection model

[0098] In some embodiments, the training method of the key point detection model includes: obtaining a sample image and a second annotated image, the second annotated image including key point coordinates corresponding to the human body area in the sample image; inputting the sample image and the second annotated image into a second pre-trained model to obtain predicted key point coordinates output by the second pre-trained model; training the second pre-trained model according to the predicted key point coordinates and the key point coordinates corresponding to the human body area in the sample image to obtain a converged key point detection model.

[0099] The specific steps include:

[0100] ① Obtain model training set

[0101] A sample image and a second annotated image are obtained as a model training set, wherein the second annotated image includes coordinates of key points corresponding to a human body region in the sample image.

[0102] In the process of obtaining the second annotated image, the key point external frame corresponding to the human body area in the sample image can be first annotated, and then the sample image can be cropped according to the key point external frame to obtain the corresponding human body image, and the human body key points are further annotated in the human body image to obtain the second annotated image. Among them, the human body key points include: nose, left and right eyes, left and right ears, left and right shoulders, left and right elbows, left and right wrists, left and right hips, left and right knees, left and right ankles, left and right feet, etc.

[0103] ②Build the initial model

[0104] The second pre-trained model as the initial model includes a backbone network, a shoulder network and a head network. It should be noted that although the structure of the second pre-trained model is the same as that of the first pre-trained model, the relevant parameters of each network are not the same. The first pre-trained model and the second pre-trained model can be constructed with reference to the prior art, which will not be elaborated in this disclosure.

[0105] ③Model training

[0106] The sample image and the second annotated image are input into the second pre-trained model to obtain the predicted key point coordinates output by the second pre-trained model, the predicted key point coordinates are compared with the true key point coordinates (the key point coordinates corresponding to the human body area in the sample image), and the corresponding loss function values ​​of the two are calculated based on the second preset loss function. When the loss function value is greater than the preset threshold, the model parameters of the second pre-trained model are adjusted, and then a new round of training is performed until the loss function value is less than or equal to the preset loss function value, so as to obtain a converged key point detection model.

[0107] In some other embodiments, the human body detection model and the key point detection model can be trained together, including: obtaining sample images and annotated images, the annotated images including the coordinates of the human body external bounding box corresponding to the sample images, and the coordinates of the key points corresponding to the leg area in the sample images; inputting the sample images and the annotated images into a pre-trained model to obtain the predicted human body external bounding box coordinates and the predicted key point coordinates output by the pre-trained model; the pre-trained model includes an initial human body detection model and an initial key point detection model; based on a first loss function and a second loss function, respectively calculating a first loss function value between the predicted human body external bounding box coordinates and the human body external bounding box coordinates, and a second loss function value between the predicted key point coordinates and the key point coordinates corresponding to the leg area; when the first loss function value and / or the second loss function value are greater than a loss threshold, adjusting the model parameters of the pre-trained model until the first loss function value and the second loss function value are less than or equal to the loss threshold, and a converged human body key point detection model is obtained.

[0108] Specifically, in the process of training the human key point detection model, firstly obtain sample images and annotated images, wherein the annotated images are annotated with the human body external frame coordinates and the leg key point coordinates based on the sample images, and the leg key point coordinates include: left hip key point coordinates, right hip key point coordinates, left knee key point coordinates, right knee key point coordinates, left foot key point coordinates and right foot key point coordinates, which are only illustrative here, and the present disclosure does not specifically limit the leg key point coordinates; a pre-training model will be constructed in advance, and the pre-training model includes an initial human body detection model and an initial key point detection model, and the structure of the pre-training model may include a backbone network, a shoulder network and a head network. Then, the sample image and the annotated image are input into the pre-training model as training sets, and the pre-training model performs calculation processing to identify the human body external bounding box coordinates and leg key point coordinates of the sample image, and obtain the predicted human body external bounding box coordinates and predicted key point coordinates output by the pre-training model, which are regarded as predicted values; further, the human body external bounding box coordinates corresponding to the annotated image and the key point coordinates corresponding to the leg area are taken as true values, and the loss function values ​​between the predicted values ​​and the true values ​​are calculated based on the first loss function and the second loss function, including: calculating the first loss function value between the predicted human body external bounding box coordinates and the human body external bounding box coordinates corresponding to the annotated image based on the first loss function, and calculating the second loss function value between the predicted key point coordinates and the leg key point coordinates in the annotated image based on the second loss function, wherein the first loss function and the second loss function may be a cross entropy loss function, a contrastive learning loss function, etc., and the present disclosure does not specifically limit this. The first loss function and the second loss function may be the same or different. Then, the first loss function value and the second loss function value are compared with the loss threshold. The loss threshold is the critical value of the loss function that characterizes the convergence of the model. If it is greater than the loss threshold, it means that the model has not converged, and if it is less than or equal to the loss threshold, it means that the model has converged. When the first loss function value is greater than the loss threshold and the second loss function value is less than or equal to the loss threshold, or when the second loss function value is greater than the loss threshold and the first loss function value is less than or equal to the loss threshold, or when the first loss function value and the second loss function value are greater than the loss threshold, it means that the model has not converged, and the model parameters of the pre-trained model are adjusted until the first loss function value and the second loss function value are both less than or equal to the loss threshold, indicating that the model has converged, and then a trained human key point detection model is obtained.

[0109] Based on the above embodiments, a converged human body detection model and a key point detection model are trained, and when the model is applied to identify key points, an image sequence is input into the human body key point detection model to obtain a human body key point coordinate sequence output by the human body key point detection model, including: inputting the image sequence into the human body detection model to obtain the human body external frame coordinates output by the human body detection model; cropping the image sequence according to the human body external frame coordinates to obtain a portrait sequence; inputting the portrait sequence into the key point detection model to obtain a human body key point coordinate sequence output by the key point detection model.

[0110] Specifically, Figure 3 As shown, Figure 3 The application diagram of the human body detection model provided in the embodiment of the present disclosure is as follows: first, the image sequence obtained in step S101 is input into the trained converged human body detection model to obtain the coordinates of the human body external bounding box output by the human body detection model, and then the images in the image sequence are cropped according to the coordinates of the human body external bounding box to obtain a portrait sequence. Figure 4 As shown, Figure 4 This is a schematic diagram of the application of the key point detection model provided in an embodiment of the present disclosure. A portrait sequence is input into the key point detection model to obtain a human body key point coordinate sequence output by the key point detection model.

[0111] The above embodiment processes the image sequence through the human key point detection model to obtain the human key point coordinate sequence corresponding to the image sequence for subsequent recognition of user actions.

[0112] S103: Input the human body key point coordinate sequence into the action recognition model to obtain the predicted action output by the action recognition model.

[0113] In some embodiments, the training method of the action recognition model includes: obtaining a sample key point coordinate sequence and annotated actions corresponding to the sample key point coordinate sequence; inputting the key point coordinate sequence and the annotated actions into a third pre-trained model, the third pre-trained model including a feature extraction module, a feature fusion module and a classification module; and training the model parameters of the third pre-trained model according to the output results of the third pre-trained model to obtain a converged action recognition model.

[0114] Specifically, the process of training the action recognition model includes the following steps:

[0115] ① Obtain model training set

[0116] Obtain the sample key point coordinate sequence and the annotated action corresponding to the sample key point coordinate sequence, wherein the annotated action is to annotate the corresponding action based on the sample key point coordinate sequence. Optionally, the sample key point coordinate sequence is a human lower limb key point coordinate sequence, including left and right hips, left and right knees, left and right ankles, and left and right feet. The annotated action includes but is not limited to the kicking action.

[0117] ②Build the initial model

[0118] The third pre-trained model as the initial model includes a feature extraction module, a feature fusion module and a classification module. The feature extraction module uses a convolutional neural network (CNN), the feature fusion module uses a machine translation (transformer) structure to fuse and associate features in time series, and the classification module uses a fully connected layer for binary classification.

[0119] ③Model training

[0120] The sample key point coordinate sequence and the labeled action corresponding to the sample key point coordinate sequence are input into the third pre-trained model, and the predicted action output by the third pre-trained model is identified. The classification module determines whether the predicted action matches the labeled action, and the loss function value of the predicted action and the labeled action is calculated based on the preset loss function. When the loss function value is greater than the preset threshold, the model parameters of the third pre-trained model are adjusted, and then a new round of training is performed until the loss function value is less than or equal to the preset loss function value, so as to obtain a converged action recognition model.

[0121] like Figure 5 As shown, Figure 5 An application schematic diagram of the action recognition model provided in the embodiment of the present disclosure, after the converged action recognition model is obtained through training by the above-mentioned training method, the human body key point coordinate sequence obtained in step S103 is input into the action recognition model to obtain the predicted action output by the action recognition model, and the predicted action indicates whether the action corresponding to the human body key point coordinate sequence matches the preset action.

[0122] In some embodiments, the process of inputting a human body key point coordinate sequence into a motion recognition model to obtain a predicted motion output by the motion recognition model includes: extracting leg key points from the human body key points; inputting the leg key points into the motion recognition model to obtain a predicted motion output by the motion recognition model.

[0123] Specifically, in order to adapt to the user's habit of opening the trunk by kicking, the present invention extracts leg key points from the identified human body key points, and then inputs the leg key points into the action recognition model to identify whether the user's leg action is a kicking action.

[0124] S104: When the predicted action matches the preset action, determine whether the foot key point in the human body key point coordinate sequence is in a preset area.

[0125] Among them, the preset action can be a kicking action.

[0126] In some embodiments, when the predicted action matches the preset action, whether the foot key points in the human body key point coordinate sequence are in a preset area is determined, including: when the predicted action matches the preset action, according to the internal and external parameters of the fisheye camera, the foot key point coordinate sequence in the image coordinate system is converted into a target coordinate sequence in the global coordinate system, the target coordinate sequence includes the foot key point coordinates in the global coordinate system; in the global coordinate system, determining whether the target coordinate sequence is in the preset area.

[0127] The internal and external parameters of the fisheye camera can be the camera internal parameters and camera external parameters of the rear-view fisheye camera, which are used for coordinate system conversion. The preset area can be the area 1m away from the left rear side of the vehicle, the right rear side of the vehicle, and the rear side of the vehicle, such as Figure 6 As shown, Figure 6 A schematic diagram of a preset area provided in an embodiment of the present disclosure. The shadow area 1m away from the left rear door, 1m away from the right rear door, and 1m away from the rear bumper of the vehicle is defined as a preset area 601.

[0128] Specifically, when the predicted action matches the kicking action, it means that the user may want to open the trunk when the kicking action is captured. To determine whether the user really wants to open the trunk, the coordinate sequence of the key points of the foot in the image coordinate system is first projected and transformed into the global coordinate system according to the internal and external parameters of the rear-view fisheye camera to determine the target coordinate sequence of the key points of the foot in the global coordinate system, thereby locating the distance between the user and the vehicle. The global coordinate system includes the preset area and the target coordinates of the key points of the foot. The positional relationship between the target coordinates and the preset area determines whether the key points of the foot are in the preset area.

[0129] In the above-mentioned embodiment, when it is recognized through the collected image sequence that the user makes a preset action, in order to avoid misjudgment, it is further determined whether the footstep key points among the human body key points are in the preset area, so that the user is in the preset area and makes the preset action to control the opening of the vehicle trunk. Compared with the related technology that uses face recognition to exclude irrelevant people from approaching the vehicle before the trunk is opened, there is no need to process the relatively private user facial information, thereby protecting the user's privacy.

[0130] S105: When the key point of the foot is in the preset area, control the vehicle trunk to open.

[0131] When the key point of the foot is in the preset area, combined with the above steps to recognize that the user makes a preset action, it can be accurately determined that the user is opening the trunk, thereby satisfying the user's expectation of controlling the opening of the vehicle trunk.

[0132] When the footstep key point is not in the preset area, it may be that the user has made a preset action unintentionally before or after getting on the vehicle. In this case, the predicted action will not be processed and the opening of the vehicle trunk will not be controlled.

[0133] In summary, the present disclosure implements a method for opening a vehicle trunk. The method first obtains an image sequence captured by a rear-view fisheye camera, and then inputs the image sequence into a human key point detection model, which includes a human detection model and a key point detection model connected in sequence, to obtain a human key point coordinate sequence output by the human key point detection model, and then inputs the human key point coordinate sequence into a motion recognition model to obtain a predicted action output by the motion recognition model. When the predicted action matches the preset action, it is determined whether the foot key points in the human key point coordinate sequence are in a preset area. If so, it means that a user in the preset area makes a preset action to open the trunk, and the vehicle trunk is controlled to open, thereby accurately identifying the user action through the human key point coordinate sequence. When the user makes the preset action, it is further determined whether the user is in the preset area, so as to eliminate the interference of actions of other irrelevant persons at a farther position of the vehicle, reduce misjudgment, and achieve precise control of the opening of the vehicle trunk.

[0134] like Figure 7 As shown, Figure 7 A vehicle trunk opening device provided in an embodiment of the present disclosure includes:

[0135] An acquisition module 701 is used to acquire an image sequence, wherein the image sequence includes multiple frames of images acquired by a target vehicle-mounted camera at different times;

[0136] The key point detection module 702 is used to input the image sequence into the human key point detection model to obtain the human key point coordinate sequence output by the human key point detection model;

[0137] The action recognition module 703 is used to input the human body key point coordinate sequence into the action recognition model to obtain the predicted action output by the action recognition model;

[0138] A judgment module 704 is used to judge whether the foot key point in the human body key point coordinate sequence is in a preset area when the predicted action matches the preset action;

[0139] The control module 705 is used to control the opening of the vehicle trunk when the key point of the foot is in a preset area.

[0140] As an optional implementation of the embodiment of the present disclosure, the human key point detection model includes a human detection model and a key point detection model connected in sequence, and the key point detection module 702 is specifically used to: input the image sequence into the human detection model to obtain the coordinates of the human body external frame output by the human detection model;

[0141] The image sequence is cropped according to the coordinates of the human body external frame to obtain a portrait sequence;

[0142] The portrait sequence is input into the key point detection model to obtain the human body key point coordinate sequence output by the key point detection model.

[0143] As an optional implementation of the embodiment of the present disclosure, the device further includes: a model training module, which is used to train the human body detection model included in the human body key point detection model, specifically used to:

[0144] Obtain a sample image and a first annotated image, where the first annotated image includes coordinates of a human body bounding box corresponding to the sample image;

[0145] Inputting the sample image and the first annotated image into the first pre-trained model to obtain the predicted bounding box coordinates output by the first pre-trained model;

[0146] The first pre-trained model is trained according to the predicted bounding box coordinates and the human body bounding box coordinates corresponding to the sample image to obtain a converged human body detection model.

[0147] As an optional implementation of the embodiment of the present disclosure, the device further includes: a model training module, which is used to train the key point detection model included in the human key point detection model, specifically used to:

[0148] Acquire a sample image and a second annotated image, wherein the second annotated image includes coordinates of key points corresponding to a human body region in the sample image;

[0149] Input the sample image and the second annotated image into the second pre-trained model to obtain the predicted key point coordinates output by the second pre-trained model;

[0150] The second pre-trained model is trained according to the predicted key point coordinates and the key point coordinates corresponding to the human body area in the sample image to obtain a converged key point detection model.

[0151] As an optional implementation of the embodiment of the present disclosure, the device further includes: a model training module, which is used to train the action recognition model, specifically used to:

[0152] Obtaining a sample key point coordinate sequence and a labeling action corresponding to the sample key point coordinate sequence;

[0153] Inputting the key point coordinate sequence and the labeled action into the third pre-trained model;

[0154] The third pre-trained model is trained according to the output actions of the third pre-trained model and the labeled actions corresponding to the sample key point coordinate sequence to obtain a converged action recognition model.

[0155] As an optional implementation of the embodiment of the present disclosure, the action recognition module 703 is specifically used to: extract a leg key point coordinate sequence from a human body key point coordinate sequence;

[0156] The leg key point coordinate sequence is input into the action recognition model to obtain the predicted action output by the action recognition model.

[0157] As an optional implementation of the embodiment of the present disclosure, the judgment module 704 is specifically used to: when the predicted action matches the preset action, convert the foot key point coordinate sequence in the image coordinate system into a target coordinate sequence in the global coordinate system according to the internal and external parameters of the fisheye camera, where the target coordinate sequence includes the foot key point coordinates in the global coordinate system;

[0158] In the global coordinate system, determine whether the target coordinate sequence is in the preset area.

[0159] like Figure 8 As shown, an embodiment of the present disclosure provides an electronic device, which includes: a processor, a memory, and a computer program stored in the memory and executable on the processor, and when the computer program is executed by the processor, each process of the vehicle trunk opening method in the above method embodiment is implemented. And the same technical effect can be achieved, so it will not be repeated here to avoid repetition.

[0160] An embodiment of the present disclosure provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, each process of the method for opening a vehicle trunk in the above method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0161] The computer readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0162] An embodiment of the present disclosure provides a vehicle, comprising: a rear-view fisheye camera; and, as described above, an opening device for a vehicle trunk, or as described above, an electronic device.

[0163] An embodiment of the present disclosure provides a computer program product, which stores a computer program. When the computer program is executed by a processor, each process of the method for opening the trunk of a vehicle in the above method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0164] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code.

[0165] In several embodiments provided by the present disclosure, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0166] In the present disclosure, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0167] In the present disclosure, memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0168] In the present disclosure, computer-readable media include permanent and non-permanent, removable and non-removable storage media. Storage media can be implemented by any method or technology to store information, and the information can be computer-readable instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. According to the definition in this article, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0169] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0170] The above are only specific embodiments of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to these embodiments herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for opening a vehicle trunk, characterized in that: include: Acquire an image sequence, wherein the image sequence includes multiple frames of images acquired by a target vehicle-mounted camera at different times; Inputting the image sequence into a human key point detection model to obtain a human key point coordinate sequence output by the human key point detection model; Inputting the human body key point coordinate sequence into an action recognition model to obtain a predicted action output by the action recognition model; In the case where the predicted action matches the preset action, determining whether the foot key point coordinate sequence in the human body key point coordinate sequence is in a preset area; When the foot key point coordinate sequence is in a preset area, the vehicle trunk is controlled to open.

2. The method according to claim 1, characterized in that The human key point detection model comprises a human detection model and a key point detection model connected in sequence, and the image sequence is input into the human key point detection model to obtain a human key point coordinate sequence output by the human key point detection model, comprising: Inputting the image sequence into the human body detection model to obtain the coordinates of the human body external bounding box output by the human body detection model; Cropping the image sequence according to the coordinates of the human body external frame to obtain a portrait sequence; The portrait sequence is input into the key point detection model to obtain a human body key point coordinate sequence output by the key point detection model.

3. The method according to claim 2, characterized in that The training method of the human body detection model comprises: Acquire a sample image and a first annotated image, where the first annotated image includes coordinates of a human body bounding box corresponding to the sample image; Inputting the sample image and the first annotated image into a first pre-trained model to obtain the predicted bounding box coordinates output by the first pre-trained model; The first pre-trained model is trained according to the predicted bounding box coordinates and the human body bounding box coordinates corresponding to the sample image to obtain a converged human body detection model.

4. The method according to claim 2, characterized in that The training method of the key point detection model includes: Acquire a sample image and a second annotated image, wherein the second annotated image includes coordinates of key points corresponding to a human body region in the sample image; Inputting the sample image and the second annotated image into a second pre-trained model to obtain the predicted key point coordinates output by the second pre-trained model; The second pre-trained model is trained according to the predicted key point coordinates and the key point coordinates corresponding to the human body area in the sample image to obtain a converged key point detection model.

5. The method according to claim 1, characterized in that: The training method of the action recognition model comprises: Obtaining a sample key point coordinate sequence and a labeling action corresponding to the sample key point coordinate sequence; Inputting the key point coordinate sequence and the annotation action into a third pre-trained model; The third pre-trained model is trained according to the output action of the third pre-trained model and the labeled action corresponding to the sample key point coordinate sequence to obtain a converged action recognition model.

6. The method according to claim 1, 2 or 5, characterized in that: The step of inputting the human body key point coordinate sequence into an action recognition model to obtain a predicted action output by the action recognition model comprises: Extracting a leg key point coordinate sequence from the human body key point coordinate sequence; The leg key point coordinate sequence is input into the action recognition model to obtain the predicted action output by the action recognition model.

7. The method according to claim 1, characterized in that When the predicted action matches the preset action, determining whether the foot key point coordinate sequence in the human body key point coordinate sequence is in a preset area includes: When the predicted action matches the preset action, converting the foot key point coordinate sequence in the image coordinate system into a target coordinate sequence in the global coordinate system according to the internal and external parameters of the target vehicle-mounted camera, wherein the target coordinate sequence includes the foot key point coordinates in the global coordinate system; In the global coordinate system, it is determined whether the target coordinate sequence is in the preset area.

8. A vehicle trunk opening device, characterized in that: include: An acquisition module, used to acquire an image sequence, wherein the image sequence includes multiple frames of images acquired by a target vehicle-mounted camera at different times; A key point detection module, used for inputting the image sequence into a human key point detection model to obtain a human key point coordinate sequence output by the human key point detection model; An action recognition module, used for inputting the human body key point coordinate sequence into an action recognition model to obtain a predicted action output by the action recognition model; A judgment module, used for judging whether a foot key point coordinate sequence in a human body key point coordinate sequence is in a preset area when the predicted action matches the preset action; The control module is used to control the opening of the vehicle trunk when the foot key point coordinate sequence is in a preset area.

9. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the method for opening a vehicle trunk as claimed in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: include: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for opening the vehicle trunk as claimed in any one of claims 1 to 7 is implemented.

11. A vehicle, characterized in that: include: Fisheye camera; And the vehicle trunk opening device as claimed in claim 8, or the electronic device as claimed in claim 9.

Images