Vehicle trunk control method, apparatus and system, and storage medium

Abstract

A vehicle trunk control method, apparatus and system, and a storage medium. The vehicle trunk control method comprises: acquiring a plurality of continuous image frames by means of an image acquisition device; according to a human body key point sequence of the plurality of image frames, determining an action recognition result corresponding to the plurality of image frames; and controlling the state of a vehicle trunk according to the action recognition result.

Description

Vehicle trunk control method, device, system and storage medium

[0001] This application claims priority to Chinese patent application No. 2023107924616, filed on June 29, 2023, entitled “Control method, device, system and storage medium for vehicle trunk”, the contents of which should be understood as incorporated into this application by reference. Technical Field

[0002] The embodiments of the present disclosure relate to, but are not limited to, the field of autonomous driving technology, and in particular to a control method, device, system, and storage medium for a vehicle trunk. Background Art

[0003] As the economy develops and the automotive industry enters the era of intelligence, users' expectations and reliance on vehicle intelligence are gradually increasing. In addition to in-cabin intelligent technologies such as Advanced Driving Assistance Systems (ADAS), Driver Monitoring Systems (DMS), and Occupant Monitoring Systems (DMS), out-of-cabin intelligent technologies, such as smart trunks, are also beginning to emerge.

[0004] The trunk can be opened and closed manually, but if both hands are occupied, the trunk cannot be opened or closed. A millimeter-wave radar sensor could be installed at the rear of the vehicle to detect the user's kicking motion. However, installing a millimeter-wave radar not only increases costs but can also cause the trunk to open if other objects pass over the sensor, potentially leading to false recognition and a potential safety hazard.

[0005] Summary of the Invention

[0006] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0007] The present disclosure provides a method, device, system and storage medium for controlling a vehicle trunk, which can improve the efficiency and accuracy of a user's control of the trunk.

[0008] In a first aspect, the present disclosure provides a method for controlling a vehicle trunk, the method comprising:

[0009] Acquire multiple continuous image frames through an image acquisition device;

[0010] Determining action recognition results corresponding to the plurality of image frames according to human body key point sequences of the plurality of image frames;

[0011] The state of the vehicle trunk is controlled according to the action recognition result.

[0012] In some embodiments, the method for determining a sequence of human body key points of a plurality of image frames includes:

[0013] Determine an action recognition detection frame based on the human body detection frames of all the image frames;

[0014] Determine the position of key points of the human body in each image frame according to the action recognition detection frame;

[0015] The human body key point sequence is determined based on the positions of the human body key points of all image frames.

[0016] In some embodiments, determining the action recognition detection frame based on the human body detection frames of all the image frames includes:

[0017] The action recognition detection frame is determined according to the minimum enclosing frame of the human body detection frames of all the image frames.

[0018] In some embodiments, the method for obtaining a human body detection frame of the image frame includes:

[0019] When there are multiple human targets in the image frame, obtaining candidate human detection frames for all human targets, where each human target corresponds to a candidate human detection frame;

[0020] According to the areas of all the candidate human detection frames and the distances between all the candidate human detection frames and the center point of the image, the human detection frame of the image frame is determined from all the candidate human detection frames.

[0021] In some embodiments, after acquiring a plurality of consecutive image frames through an image acquisition device, the method further includes:

[0022] The positions of the human key points in each of the image frames are identified by a pre-trained human key point detection model, wherein the positions of the human key points in all the image frames constitute the human key point sequence, and the training method of the human key point detection model includes:

[0023] Training the initial model based on the heat map, and obtaining a first model after the initial model converges;

[0024] Keeping the weight parameters of the first model fixed except for the heat map branch, replacing the heat map branch of the first model with the regression branch to obtain a second model;

[0025] The second model is trained to obtain the human body key point detection model.

[0026] In some embodiments, determining the action recognition results corresponding to the plurality of image frames based on the human body key point sequences of the plurality of image frames includes:

[0027] Determining motion information corresponding to all the image frames according to the human body key point sequence;

[0028] Determine skeleton information corresponding to each of the image frames according to the human body key point sequence;

[0029] The action recognition result is determined according to the motion information and skeleton information corresponding to all the image frames.

[0030] In some embodiments, determining motion information corresponding to the plurality of image frames according to the human body key point sequence includes:

[0031] Determining the human body key point change information corresponding to the two adjacent image frames according to the difference in the positions of the human body key points between the two adjacent image frames;

[0032] The motion information is determined based on all the human body key point change information.

[0033] In some embodiments, determining the skeleton information corresponding to each image frame according to the human body key point sequence includes:

[0034] For each of the image frames:

[0035] Determine skeleton vectors of all human body parts respectively according to the human body key point sequence;

[0036] The skeleton information corresponding to the image frame is determined according to all the skeleton vectors of the image frame.

[0037] In a second aspect, an embodiment of the present disclosure provides a control device for a vehicle trunk, the device comprising an acquisition module, a determination module, and a control module;

[0038] The acquisition module is configured to acquire a plurality of continuous image frames through an image acquisition device;

[0039] The determination module is configured to determine action recognition results corresponding to the plurality of image frames based on a human body key point sequence of the plurality of image frames;

[0040] The control module is configured to control the state of the vehicle trunk according to the action recognition result.

[0041] In a third aspect, an embodiment of the present disclosure provides a vehicle trunk control system, the system comprising an image acquisition device and a processor:

[0042] The image acquisition device acquires a plurality of continuous image frames;

[0043] The processor determines action recognition results corresponding to the multiple image frames based on the human body key point sequences of the multiple image frames; and controls the state of the vehicle trunk according to the action recognition results.

[0044] In a fourth aspect, an embodiment of the present disclosure provides an electronic device comprising a processor and a memory storing a computer program that can be run on the processor; the processor is configured to implement a vehicle trunk control method as described in any one of the first aspects above when executing the program.

[0045] In a fifth aspect, an embodiment of the present disclosure provides a non-volatile computer-readable storage medium storing one or more program instructions, wherein the one or more program instructions are used to execute the vehicle trunk control method as described in any one of the first aspects above.

[0046] The vehicle trunk control method provided in this embodiment obtains multiple continuous image frames through an image acquisition device; determines the action recognition results corresponding to the multiple image frames based on the human body key point sequence of the multiple image frames; controls the trunk status of the vehicle based on the action recognition results, thereby improving the efficiency and accuracy of the user's control of the trunk; the image acquisition device can be an existing device on the vehicle, and there is no need to install additional equipment (such as an image acquisition device or a sensor), thereby reducing the cost of the vehicle.

[0047] Other features and advantages of the present disclosure will be described in the following description or understood through implementation of the present disclosure. Other advantages of the present disclosure can be realized and obtained through the solutions described in the description and the drawings.

[0048] Still other aspects will become apparent upon reading and understanding the accompanying drawings and detailed description.

[0049] Summary of the Figures

[0050] The accompanying drawings are used to provide an understanding of the technical solution of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the technical solution of the present disclosure and do not constitute a limitation to the technical solution of the present disclosure.

[0051] FIG1 is a flow chart of a method for controlling a vehicle trunk provided by an embodiment of the present disclosure;

[0052] FIG2 is a flow chart of a method for determining a human body key point sequence provided by an embodiment of the present disclosure;

[0053] FIG3 is a schematic diagram of an action recognition detection frame provided by an embodiment of the present disclosure;

[0054] FIG4 is a flow chart of a method for training a human key point detection model according to an embodiment of the present disclosure;

[0055] FIG5 is a flow chart of a method for motion recognition provided by an embodiment of the present disclosure;

[0056] FIG6 is a schematic diagram of key points of a human body provided by an embodiment of the present disclosure;

[0057] FIG7 is a flow chart of a method for controlling a vehicle trunk provided by an optional embodiment of the present disclosure;

[0058] FIG8 is a structural block diagram of a vehicle trunk control device provided by an embodiment of the present disclosure;

[0059] FIG9 is a structural block diagram of an electronic device provided by an embodiment of the present disclosure.

[0060] Details

[0061] In order to more clearly understand the purpose, technical solutions and advantages of the present disclosure, the present disclosure is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0062] Unless otherwise defined, the technical or scientific terms involved in this disclosure should have the general meaning understood by people with ordinary skills in the technical field to which this application belongs. In this application, the words "one", "a", "the", "these" and the like do not indicate a limit on quantity, and they can be singular or plural. The terms "include", "comprise", "have" and any variations thereof involved in this application are intended to cover non-exclusive inclusion; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships can exist. For example, "A and / or B" can mean: A exists alone, A and B exist, and B exists alone. Generally, the character " / " indicates that the objects in the preceding and following relationship are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0063] The embodiment of the present application provides a method for controlling a vehicle trunk, as shown in FIG1 , comprising the following steps:

[0064] Step S101: Acquire a plurality of continuous image frames through an image acquisition device.

[0065] Among them, the image acquisition device can be installed at the rear of the vehicle. Depending on different vehicle models and needs, other cameras installed in the vehicle can also be used.

[0066] In this embodiment, the image acquisition device can be a device of the vehicle itself and does not require additional installation. For example, it can be a reversing image camera built into the trunk of the vehicle.

[0067] The image frame capture range can be set as needed to ensure that at least the key points of the action required to complete the target are visible in the image. For example, if the action used to control the trunk is set to a leg sweep, at least the lower body of the person, such as the hips and below, must be visible.

[0068] Similarly, the moment the image acquisition device begins capturing data can be customized based on your needs. For example, sensing a car key within a preset area will activate the trunk control function through motion recognition. The system then detects the human body and key points, using these key points to locate the person. Only when the person is within the preset location will the trunk opening action be recognized.

[0069] Step S102 : determining action recognition results corresponding to the multiple image frames according to the human body key point sequences of the multiple image frames.

[0070] When performing action recognition based on multiple image frames, the positions of the human body key points in each image frame can be identified first to determine the human body posture in the corresponding image frame, and then the positions of the human body key points of all image frames are spliced ​​and fused in time sequence to obtain a human body key point sequence corresponding to multiple image frames, so as to recognize the human body posture of multiple image frames based on the human body key point sequence and obtain a complete action process, that is, the action recognition results of multiple image frames.

[0071] The human keypoint sequence is composed of all keypoint positions in all image frames. For example, if an image frame has 20 human keypoints, the positions of the human keypoints are obtained and then a group of human keypoint positions is formed based on the human keypoint positions. In the case of 32 frames of images, 32 groups of human keypoint positions form a human keypoint sequence.

[0072] Step S103: controlling the state of the vehicle trunk according to the action recognition result.

[0073] Among them, the actions for controlling the trunk can be factory set or set by the user. The actions can include at least one of the upper body movements and lower body movements; the upper body movements can include waving, drawing circles, and clapping; the lower body movements can include kicking, squatting, bending over, etc.

[0074] When the motion recognition results corresponding to multiple image frames match the preset motion, the trunk can be controlled to open or close. If they do not match, the trunk remains in its original state. The trunk opening and closing can be set to the same motion or different motions.

[0075] Through the above steps S101 to S103, this embodiment collects multiple image frames near the vehicle based on the image acquisition device, and then performs motion recognition based on the multiple image frames, and realizes control of the trunk according to the motion recognition results; the image acquisition device can be an existing device on the vehicle, and there is no need to install additional equipment (such as image acquisition equipment or sensors), which reduces the cost of the vehicle. The efficiency and accuracy of the user's control of the trunk are also improved through motion recognition.

[0076] After obtaining multiple image frames, if the human body images are obtained by cropping each image frame according to the human body detection frame of each image frame, and then action recognition is performed based on the human body images, since the human body range of each image frame is different and the coordinate system is also different, the multiple human body images obtained based on the multiple image frames will lose the relative position relationship between the different image frames, thereby losing the motion trajectory information, resulting in a decrease in the accuracy of action recognition. Therefore, in some embodiments, the coordinate systems of the multiple image frames can be unified, and action recognition can be performed within a common range. Figure 2 is a flow chart of the method for determining the human body key point sequence provided by an embodiment of the present application. As shown in Figure 2, the method includes the following steps:

[0077] Step S201 : determining an action recognition detection frame based on human body detection frames of all image frames.

[0078] Step S202 : determining the position of key points of the human body in each image frame according to the action recognition detection frame.

[0079] Step S203: determining the human body key point sequence based on the human body key point positions of all image frames.

[0080] When identifying a human body in an image, you can first determine the human body detection frame in the image frame, that is, the area in the image where the human body is located, and then identify and detect the human body based on the area in the human body detection frame. Therefore, each image frame has a corresponding human body detection frame. For example, the human body detection frame can be expressed in the form of two-dimensional coordinates, that is, (left, top, bottom, right). The Faster Regional Convolutional Neural Network (Faster RCNN) algorithm or the YOLO algorithm can be used. Taking the YOLOV5 algorithm as an example, based on the low computing power of the vehicle, the network model is compressed to detect the target quickly and accurately.

[0081] In this embodiment, after a human body detection frame is determined in each image frame, an action recognition detection frame that can include all human body detection frames is determined based on all human body detection frames to preserve the relative positional relationship between postures in multiple image frames and the continuity of the action.

[0082] After determining the action recognition detection frame, the positions of human key points are determined in each image frame based on the action recognition detection frame, and a human key point sequence is obtained based on the positions of the human key points in all image frames. For example, after determining the human detection frames of all image frames, the action recognition detection frame is determined based on the human detection frames of all image frames. In each image frame, the image frame is cropped and scaled based on the action recognition detection frame to obtain multiple image frames of uniform size, thereby obtaining a human key point sequence consisting of the positions of human key points in the multiple image frames of uniform size.

[0083] Through the above steps S201 and S203, this embodiment determines a human key point sequence based on the action recognition detection frame obtained based on all human detection frames, and can retain the relative position relationship and action trajectory between actions in multiple image frames, thereby improving the accuracy of action recognition.

[0084] In some embodiments, the action recognition detection frame is determined based on the minimum enclosing frame of the human body detection frames of all image frames. Since multiple image frames record the changes in human body movements, the movements of each image frame are different, and the position of the human body detection frame of each image frame is also different. After acquiring all image frames, the embodiment of the present application determines the topmost boundary, bottommost boundary, leftmost boundary and rightmost boundary of all human body detection frames from the human body detection frames of all image frames, and recomposes a new human body detection frame from the determined boundaries, and the new human body detection frame is the minimum enclosing frame. As shown in Figure 3, there are 4 image frames on the left, the dotted line is the human body detection frame, and the solid line frame inside the right image is the minimum enclosing frame of the human body detection frame in the 4 image frames on the left. For example, after obtaining the human body detection frames of all image frames, all image frames with human body detection frames can be superimposed to obtain the superposition result of multiple human body detection frames, and the position and range of the minimum enclosing frame are determined based on the superposition result. For example, the top, bottom, left, and right boundaries of the minimum bounding box are determined based on the outer boundaries of the superimposed results of multiple human detection frames, and then, based on the range determined by the boundaries, an action recognition detection box that exactly contains all the human detection frames is obtained. In this embodiment, the action recognition detection box is determined based on the minimum bounding box of the human detection frames in all image frames, and action recognition is performed based on the area of ​​the action recognition detection box. This can preserve the continuity of the action across multiple image frames and improve the accuracy of action recognition.

[0085] In some embodiments, there may be multiple human targets in an image frame, so multiple candidate human detection frames can be obtained. At this time, the multiple candidate human detection frames can be screened to obtain a human detection frame to improve the accuracy of action recognition. For example, the method for obtaining the human detection frame of an image frame includes: when there are multiple human targets in an image frame, obtaining the candidate human detection frames of all human targets, that is, each human target corresponds to a human detection frame, and the human detection frame of each human target can be used as a candidate human detection frame. Then, based on the area of ​​all candidate human detection frames and the distance of all candidate human detection frames from the center point of the image, the human detection frame of the image frame is determined among all candidate human detection frames. Optionally, a candidate human detection frame whose distance from the center point of the image is less than a preset distance and whose area is greater than a preset area can be selected as the human detection frame, thereby obtaining the main person located in the center of the image and improving the accuracy of action recognition.

[0086] In some embodiments, after obtaining multiple image frames, the positions of human key points in the multiple image frames can be identified using a pre-trained human key point detection model, so as to form a human key point sequence based on the positions of the human key points in the multiple image frames. FIG4 is a flow chart of a method for training a human key point detection model provided in an embodiment of the present application, and the method includes the following steps:

[0087] Step S401: Train the initial model (i.e., an untrained human key point detection model) based on the heat map, and obtain a first model after the initial model converges.

[0088] The initial network includes two parts: a backbone network and a prediction network; the prediction network only includes a heat map branch;

[0089] The heatmap branch can output a heatmap for each keypoint. For example, if there are N keypoints in an image frame, N heatmaps will be output. Each pixel in the heatmap represents the probability of the keypoint existing at that pixel location. Through post-processing, the pixel location with a probability greater than a certain threshold is the keypoint location. The heatmap branch is more robust to large-scale motion and image occlusion.

[0090] Step S402: Keep the weight parameters of the first model except the heat map branch fixed, replace the heat map branch of the first model with the regression branch, and obtain the second model.

[0091] The regression branch directly outputs a vector representing the position and latitude of a key point in the human body, expressed as (x, y, vis), where x and y represent the position of the key point and vis represents visibility. A value of 0 indicates that the key point does not exist in the image. The advantage of this regression branch is that it requires less computation and can better preserve the topological structure of the human body.

[0092] Step S403: training the second model to obtain a human key point detection model.

[0093] The human key point detection model consists of two parts: a backbone network and a prediction network. The backbone network can be a residual network (ResNet, HrNet, etc.); the prediction network includes a heat map branch and a regression branch. For example, the regression branch is usually composed of several fully connected layers, while the heat map branch is usually composed of several upsampling or deconvolution layers plus sigmoid layers.

[0094] The human key point detection model described in the embodiment of the present application uses a heat map branch and a regression branch to locate the human key points, so that the process of locating the human key points has the advantages of the above two branches.

[0095] Through the above steps S401 to S403, in this embodiment, the heat map is combined with regression, and a human key point detection model is obtained after training to save computing power in the action recognition process and improve the accuracy of action recognition.

[0096] In some embodiments, FIG5 is a flow chart of a method for motion recognition provided by an embodiment of the present application. As shown in FIG5 , the method includes the following steps:

[0097] Step S501 : determining motion information corresponding to a plurality of image frames according to a human body key point sequence.

[0098] The motion information in this embodiment refers to the human body motion process obtained through multiple image frames. Since each frame of image can obtain a human body posture, the complete motion process can be obtained according to the posture changes in different frame images.

[0099] Step S502, determining the skeleton information corresponding to each image frame according to the human body key point sequence;

[0100] The human key point sequence contains the positions of the human key points for each image frame. Therefore, the skeleton information of the corresponding image frame can be calculated based on the positions of the human key points in each image frame. In this embodiment, the skeleton information refers to the relative positions of multiple human key points in a single frame image, and can be represented by vectors consisting of multiple sets of two adjacent human key points.

[0101] Step S503: determining an action recognition result according to the motion information and the skeleton information.

[0102] In this embodiment, the action recognition process is implemented through Graph Convolution Neural Networks (GCNNs for short).

[0103] Through the above steps S501 to S503, action recognition is achieved based on both motion information and skeleton information, which can improve the accuracy of action recognition.

[0104] In some embodiments, a sequence of key points can be represented as a three-dimensional vector keypoints with the dimension [T, C, 3]. T represents the length of the key point sequence; C represents the number of key points; and 3 represents the dimensions, including x, y, and vis. This three-dimensional vector is input into a graph convolutional neural network to obtain the classification result, i.e., the action recognition result.

[0105] For example, the method for determining motion information includes: since the human key point sequence contains the human key point position of each image frame, after obtaining the human key point sequence, the human key point change information corresponding to the two adjacent image frames can be determined in sequence according to the difference in the key point position between the two adjacent image frames, and then the motion information is determined based on the multiple human key point change information. The two image frames in this embodiment can be two adjacent image frames or two non-adjacent image frames. In the process of obtaining the multiple human key point change information, the interval between the two image frames can be the same or different. For example, the first human key point change information can be obtained by subtracting the first frame from the second frame, the second human key point change information can be obtained by subtracting the second frame from the third frame, the third human key point change information can be obtained by subtracting the third frame from the fourth frame, and so on. Then, the first human key point change information, the second human key point change information, the third human key point change information, and the other human key point change information together constitute the motion information. Optionally, when the human key point change information is represented by motion, motion=[T t+1 ,C,3]-[T t ,C,3], where the subscript T refers to the (t+1)th frame or the tth frame in the human key point sequence. In this embodiment, the motion information contained in multiple image frames is obtained by calculating the difference between the positions of the human key points of two image frames. This motion information can reflect the user's motion trajectory and help improve the accuracy of motion recognition.

[0106] For example, the method for determining skeleton information includes: for each image frame, determining the skeleton vectors of all human body parts respectively according to the human body key point sequence, and then determining the skeleton information corresponding to the image frame according to all the skeleton vectors of the image frame. Since the human body key point sequence includes the position coordinates of each human body key point in each frame of the image, the skeleton vector can be calculated based on the position coordinates of the human body key point of each human body part, wherein the division of human body parts can be set according to needs, which can be half body or whole body, and the whole body includes head, torso, limbs, etc. Each human body part corresponds to a skeleton vector, and the skeleton vectors of all human body parts together constitute the skeleton information. The posture of each image frame can be obtained through the skeleton information, which helps to improve the accuracy of action recognition.

[0107] FIG6 is a schematic diagram of the key points of the human body provided in an embodiment of the present application. As shown in FIG6 , the key points of the human body in this embodiment include the head, torso, limbs, feet, etc., and each key point of the human body has a corresponding numerical serial number. For example, the starting point of the head skeleton is 9 and the end point is 8. If the skeleton information is represented by skeleton, the head skeleton is represented by skeleton_9. For example, the vector corresponding to the head skeleton is represented by skeleton_9 = [T, C8, 3] - [T, C9, 3)]. The skeleton vectors of multiple parts of the human body are stacked to obtain the final skeleton information skeleton.

[0108] After obtaining the human body key point sequence keypoints, motion information motion and skeleton information skeleton, [keypoints, motion, skeleton] can be input into the graph convolutional neural network for action recognition.

[0109] The following is an optional embodiment of the present application. FIG7 is a flow chart of a method for controlling a vehicle trunk provided in an optional embodiment of the present application. As shown in FIG7 , the method includes the following steps:

[0110] Step S701, collecting multiple image frames through a camera at the rear of the vehicle;

[0111] Step S702: Perform human body detection on each image frame to obtain multiple human body detection frames, where each image frame corresponds to a human body detection frame;

[0112] Step S703, performing human body key point position detection in each human body detection frame;

[0113] Step S704, cropping and scaling the multiple image frames according to the minimum bounding box of all human body detection frames to obtain a sequence of human body key points;

[0114] Step S705, normalizing the human body key point sequence;

[0115] Step S706, obtaining motion information between multiple image frames and skeleton information of each image frame according to the human body key point sequence;

[0116] Step S707 , performing action recognition based on the motion information and skeleton information, and controlling the vehicle trunk based on the action recognition result.

[0117] Through the above steps S701 to S707, images near the vehicle are collected based on the image acquisition device on the vehicle, and then action recognition is performed based on multiple image frames. The trunk is controlled according to the action recognition results, thereby improving the efficiency and accuracy of the user's control of the trunk.

[0118] The steps shown in the above processes or the flowcharts in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be performed in an order different from that shown here.

[0119] This embodiment also provides a vehicle trunk control device configured to implement the aforementioned embodiments and alternative implementations. Details already described are omitted for clarity. Terms such as "module," "unit," and "subunit" used below may refer to a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are implemented using software, implementation using hardware, or a combination of software and hardware, is also possible and contemplated.

[0120] FIG8 is a structural block diagram of a vehicle trunk control device according to an embodiment of the present application. As shown in FIG8 , the device includes an acquisition module 81 , a determination module 82 , and a control module 83 ;

[0121] The acquisition module 81 is configured to acquire a plurality of continuous image frames through an image acquisition device;

[0122] The determining module 82 is configured to determine the action recognition results corresponding to the plurality of image frames according to the human body key point sequences of the plurality of image frames;

[0123] The control module 83 is configured to control the state of the vehicle trunk according to the action recognition result.

[0124] In this embodiment, the acquisition module 81 captures images near the vehicle based on the image acquisition device on the vehicle, the determination module 82 performs action recognition based on multiple image frames, and the control module 83 controls the trunk according to the action recognition results, thereby improving the efficiency and accuracy of the user's control of the trunk.

[0125] For example, the determination module 82 is further configured to determine an action recognition detection frame based on the human body detection frames of all the image frames; and obtain the human body key point sequence in the image region determined by the action recognition detection frame. Optionally, the action recognition detection frame is determined based on the minimum bounding box of the human body detection frames of all the image frames.

[0126] For example, the determination module 82 is further configured to determine a human body detection frame: when there are multiple human targets in the image frame, obtain candidate human body detection frames for all human targets; and determine the human body detection frame of the image frame among the multiple candidate human body detection frames based on the areas of the multiple candidate human body detection frames and the distances between the multiple candidate human body detection frames and the center point of the image.

[0127] In this embodiment, a pre-trained human key point detection model is used to identify the position of the human key points in each image frame, wherein the human key point positions in multiple image frames constitute the human key point sequence. The device also includes a training module, which is configured to train the initial model based on the heat map to obtain a first model after the initial model converges; keep the weight parameters outside the heat map branch in the first model fixed, replace the heat map branch of the first model with a regression branch to obtain a second model; train the second model to obtain the human key point detection model.

[0128] For example, the control module 83 is configured to determine motion information corresponding to multiple image frames based on the human key point sequence; determine skeleton information for each image frame based on the human key point sequence; and determine the action recognition result based on the motion information and the skeleton information. Specifically, based on the difference in human key point positions between two image frames, human key point change information corresponding to the two image frames is determined; and based on the multiple human key point change information, the motion information is determined. Specifically, for each image frame, skeleton vectors for multiple human body parts are determined based on the human key point sequence; and the skeleton information is determined based on all the skeleton vectors.

[0129] Each of the above modules can be a functional module or a program module, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0130] The present application also provides a control system for a vehicle trunk, which includes an image acquisition device and a processor: the image acquisition device obtains multiple continuous image frames; the processor determines the action recognition results corresponding to the multiple image frames based on the human body key point sequence of the multiple image frames; and controls the trunk status of the vehicle based on the action recognition results.

[0131] This embodiment further provides an electronic device, as shown in FIG9 , including a memory 901 and a processor 902 . The memory 901 stores a computer program, and the processor 902 is configured to run the computer program to execute the steps in any of the above method embodiments.

[0132] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0133] Optionally, in this embodiment, the processor may be configured to execute the following steps through a computer program:

[0134] S1, acquiring multiple continuous image frames through an image acquisition device.

[0135] S2: Determine action recognition results corresponding to the multiple image frames according to the human body key point sequences of the multiple image frames.

[0136] S3: Control the trunk status of the vehicle according to the action recognition result.

[0137] For examples in this embodiment, reference may be made to the examples described in the above embodiments and optional implementation modes, and will not be repeated in this embodiment.

[0138] In addition, in conjunction with the vehicle trunk control method provided in the above embodiments, this embodiment may also provide a computer-readable storage medium for implementation. The storage medium stores one or more programs, each of which is executable by one or more processors. When executed by the processor, the program implements any of the methods in the above embodiments.

[0139] The embodiments described herein are only used to explain this application, not to limit it. Based on the embodiments provided in this application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0140] The accompanying drawings are merely examples or embodiments of the present application. A person skilled in the art can apply the present application to other similar situations based on these drawings without inventive effort. Furthermore, it is understood that although the work involved in the development process may be complex and lengthy, certain design, manufacturing, or production changes based on the technical content disclosed in this application are merely routine technical means for a person skilled in the art and should not be considered as insufficient disclosure of the present application.

[0141] The term "embodiment" as used in this application refers to features, structures, or characteristics described in connection with the embodiment that can be included in at least one embodiment of this application. The appearance of this phrase in multiple places in the specification does not necessarily refer to the same embodiment, nor does it imply that the embodiments are mutually exclusive and independent or optional. It is understood, either explicitly or implicitly, by those skilled in the art that the embodiments described in this application can be combined with other embodiments unless there is a conflict.

[0142] The above-described embodiments merely represent several implementation methods of the present application, and although the descriptions are relatively detailed, they should not be construed as limiting the scope of the patent protection. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and such modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

[0143] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all components may be implemented as software executed by a processor, such as a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As is well known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable, and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media generally embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

Claims

1. A method for controlling a vehicle trunk, comprising: Acquire a plurality of continuous image frames through an image acquisition device; Determine action recognition results corresponding to the plurality of image frames according to the human body key point sequences of the plurality of image frames; The state of the vehicle trunk is controlled according to the action recognition result.

2. The vehicle trunk control method according to claim 1, wherein: The method for determining a human body key point sequence of a plurality of image frames comprises: Determine an action recognition detection frame according to the human body detection frames of all the image frames; Determine the position of key points of the human body in each image frame according to the action recognition detection frame; The human body key point sequence is determined based on the human body key point positions of all image frames.

3. The vehicle trunk control method according to claim 2, wherein: Determining the action recognition detection frame according to the human body detection frames of all the image frames comprises: The action recognition detection frame is determined according to the minimum enclosing frame of the human body detection frames of all the image frames.

4. The vehicle trunk control method according to claim 2, wherein: The method for acquiring the human body detection frame of the image frame includes: When there are multiple human targets in the image frame, obtaining candidate human detection frames of all human targets, each human target corresponding to a candidate human detection frame; According to the areas of all the candidate human body detection frames and the distances between all the candidate human body detection frames and the center point of the image, the human body detection frame of the image frame is determined among all the candidate human body detection frames.

5. The vehicle trunk control method according to claim 2, after acquiring a plurality of continuous image frames by an image acquisition device, the method further comprises: The position of the human key point in each of the image frames is identified by a pre-trained human key point detection model, wherein the positions of the human key points in all the image frames constitute the human key point sequence, and the training method of the human key point detection model includes: Training the initial model based on the heat map, and obtaining a first model after the initial model converges; Keeping the weight parameters of the first model except the heat map branch fixed, replacing the heat map branch of the first model with the regression branch, to obtain a second model; The second model is trained to obtain the human body key point detection model.

6. The vehicle trunk control method according to claim 1, wherein: Determining the action recognition results corresponding to the plurality of image frames according to the human body key point sequences of the plurality of image frames comprises: Determine motion information corresponding to all the image frames according to the human body key point sequence; Determine skeleton information corresponding to each of the image frames according to the human body key point sequence; The action recognition result is determined according to the motion information and the skeleton information corresponding to all the image frames.

7. The vehicle trunk control method according to claim 6, wherein: Determining the motion information corresponding to the plurality of image frames according to the human body key point sequence comprises: Determining the human body key point change information corresponding to two adjacent image frames according to the difference in the positions of the human body key points between two adjacent image frames; The motion information is determined according to all the key point change information of the human body.

8. The vehicle trunk control method according to claim 6 or 7, wherein: Determining the skeleton information corresponding to each of the image frames according to the human body key point sequence comprises: For each of the image frames: Determine skeleton vectors of all human body parts respectively according to the human body key point sequence; The skeleton information corresponding to the image frame is determined according to all the skeleton vectors of the image frame.

9. A control device for a vehicle trunk, the device comprising an acquisition module, a determination module and a control module; The acquisition module is configured to acquire a plurality of continuous image frames through an image acquisition device; The determination module is configured to determine the action recognition results corresponding to the plurality of image frames according to the human body key point sequences of the plurality of image frames; The control module is configured to control the state of the vehicle trunk according to the action recognition result.

10. A control system for a vehicle trunk, wherein: The system comprises an image acquisition device and a processor: The image acquisition device acquires a plurality of continuous image frames; The processor determines action recognition results corresponding to the multiple image frames according to the human body key point sequences of the multiple image frames; and controls the state of the vehicle trunk according to the action recognition results.

11. A non-transitory computer-readable storage medium storing one or more program instructions, wherein the one or more program instructions are used to execute the vehicle trunk control method according to any one of claims 1 to 8.

12. An electronic device comprising a processor and a memory storing a computer program executable on the processor, wherein: When the processor executes the program, the steps of the vehicle trunk control method according to any one of claims 1 to 8 are implemented.

Images