Game interaction control methods, devices, computer equipment, and storage media

Abstract

This application relates to a game interaction control method, apparatus, computer device, and storage medium. The method includes: responding to a game interaction request triggered on a vehicle's in-vehicle platform, processing and obtaining the target detection result of a received video frame; if the target detection result indicates that the video frame contains multiple candidate objects, then identifying the pose recognition result of each candidate object based on the target tracking data of each candidate object; setting the candidate object corresponding to the target pose recognition result that matches a preset pose in the pose recognition result as an interaction object, and identifying the action intent information of the interaction object based on the target tracking data of the interaction object; mapping the action intent information to game interaction control information of the in-vehicle game corresponding to the game interaction request, so as to control the game character in the in-vehicle game to perform corresponding actions through the game interaction control information. This method can improve the accuracy of controlling in-vehicle games through the actions of interaction objects.

Description

Technical Field

[0001] This application relates to the field of human-computer interaction and intelligent control technology, and in particular to a game interaction control method, device, computer equipment, storage medium and computer program product. Background Technology

[0002] With the development of intelligent vehicle technology and human-machine interaction systems, human motion detection and interactive control based on visual recognition has gradually become an important research direction for in-vehicle entertainment and auxiliary control systems.

[0003] Existing in-vehicle game and human-computer interaction control typically relies on simple gesture recognition for game interaction, resulting in a limited range of interaction methods. However, the real-world in-vehicle environment is complex and variable, with factors such as changing lighting, viewing angles, and human occlusion. Using traditional visual recognition algorithms in such environments leads to low accuracy in recognizing human movements, resulting in low accuracy in controlling in-vehicle games through human gestures. Summary of the Invention

[0004] Therefore, it is necessary to provide a game interaction control method, device, computer equipment, computer-readable storage medium, and computer program product that can improve the accuracy of controlling in-vehicle games through human body movements, in order to address the above-mentioned technical problems.

[0005] Firstly, this application provides a game interaction control method. The method includes:

[0006] In response to a game interaction request triggered on the vehicle's in-vehicle platform, target detection processing is performed on the received video frame to obtain the target detection result of the video frame; the video frame is obtained by capturing the game interaction area of ​​the vehicle through the vehicle's camera device;

[0007] If the target detection result indicates that the video frame contains multiple candidate objects, then pose recognition processing is performed on the target tracking data of each candidate object to obtain the pose recognition result of each candidate object; the target tracking data is obtained by performing target tracking processing on the video frame;

[0008] If a target posture recognition result matching a preset posture is detected in the posture recognition results, the candidate object corresponding to the target posture recognition result is set as an interaction object, and the target tracking data of the interaction object is processed for action intent recognition to obtain the action intent information of the interaction object.

[0009] The action intent information is mapped to the game interaction control information of the in-vehicle game corresponding to the game interaction request, so as to control the game character in the in-vehicle game to perform the corresponding action through the game interaction control information.

[0010] In one embodiment, target detection processing is performed on the received video frame to obtain the target detection result of the video frame, including:

[0011] Obtain the brightness information of the target region in the video frame;

[0012] Based on the brightness information, the mean brightness and skewness coefficient of the target area are obtained;

[0013] If both the mean brightness and the skewness coefficient meet the preset illumination conditions, then target detection processing is performed on the video frame to obtain the target detection result of the video frame.

[0014] In one embodiment, target detection processing is performed on the video frame to obtain the target detection result of the video frame, including:

[0015] The video frame is processed in both the channel and spatial dimensions using an object detection model to obtain the target feature information of the video frame. The object detection model is obtained by optimizing the feature processing module of the base model through a multi-branch attention mechanism.

[0016] Using the target detection model, based on the target feature information, target detection processing is performed on the video frame to obtain the bounding box information of the candidate object in the video frame;

[0017] Based on the bounding box information, key point recognition processing is performed on the candidate objects in the video frame to obtain the key point information of the candidate objects;

[0018] The target detection result is obtained based on the bounding box information and the key point information.

[0019] In one embodiment, the target tracking data of the interactive object is processed for action intent recognition to obtain the action intent information of the interactive object, including:

[0020] The key point information in the target tracking data of the interactive object in consecutive multi-frames is processed to identify the first action intention for the first action, so as to obtain the first action intention information of the interactive object.

[0021] If the first action intent information representation does not conform to the first action, then the key point information of multiple consecutive frames is processed for action intent recognition of the second action to obtain the second action intent information of the interactive object.

[0022] If the second action intent information representation does not conform to the second action, then the key point information of multiple consecutive frames is processed for action intent recognition of the third action to obtain the third action intent information of the interactive object.

[0023] The first action, the second action, and the third action are different.

[0024] In one embodiment, the key point information of multiple consecutive frames is processed for action intent recognition of a first action to obtain the first action intent information of the interactive object, including:

[0025] Obtain the hip position information corresponding to the hip key point and the knee position information corresponding to the knee key point from the key point information;

[0026] Based on the positional difference between the hip position information and the knee position information, the height difference information between the hip key point and the knee key point is obtained;

[0027] Based on the height difference information of multiple consecutive frames, it is determined whether the interactive object triggers the first action, and the first action intent information of the interactive object is obtained.

[0028] In one embodiment, the key point information of multiple consecutive frames is subjected to intent recognition processing for the third action to obtain the third action intent information of the interactive object, including:

[0029] Obtain the ankle position information corresponding to the ankle key points in the key point information;

[0030] Based on the knee position information of multiple consecutive frames, the knee displacement information of the knee key points is obtained;

[0031] Based on the ankle position information of multiple consecutive frames, the ankle displacement information of the key ankle points is obtained;

[0032] Based on the knee displacement information and the ankle displacement information, it is determined whether the interactive object triggers the third action, and the third action intention information of the interactive object is obtained.

[0033] In one embodiment, the key point information of multiple consecutive frames is subjected to intent recognition processing for the second action to obtain the second action intent information of the interactive object, including:

[0034] Obtain the left shoulder position information corresponding to the left shoulder key point and the right shoulder position information corresponding to the right shoulder key point from the key point information;

[0035] Based on the left shoulder position information and the right shoulder position information, the shoulder line of the interactive object is obtained;

[0036] Determine the angle between the shoulder line and the target bounding box line in the target tracking data of the interactive object;

[0037] Based on the included angle of multiple consecutive frames, it is determined whether the interactive object triggers the second action, thereby obtaining the second action intention of the interactive object.

[0038] In one embodiment, pose recognition processing is performed on the target tracking data of each candidate object to obtain the pose recognition result of each candidate object, including:

[0039] Extract the bounding box region for each candidate object from the target tracking data of each candidate object;

[0040] By using a pose classifier for the preset pose, pose classification processing is performed on the bounding box region of each candidate object to obtain the pose recognition result of each candidate object; wherein, the pose recognition result is used to characterize whether the pose of the candidate object matches the preset pose.

[0041] Secondly, this application also provides a game interaction control device. The device includes:

[0042] The target detection module is used to respond to a game interaction request triggered on the vehicle's in-vehicle platform, perform target detection processing on the received video frame, and obtain the target detection result of the video frame; the video frame is obtained by capturing the game interaction area of ​​the vehicle through the vehicle's camera device;

[0043] The pose recognition module is used to perform pose recognition processing on the target tracking data of each candidate object if the target detection result indicates that the video frame contains multiple candidate objects, so as to obtain the pose recognition result of each candidate object; the target tracking data is obtained by performing target tracking processing on the video frame;

[0044] The action intent recognition module is used to, if it detects that there is a target posture recognition result in the posture recognition result that matches a preset posture, set the candidate object corresponding to the target posture recognition result as an interaction object, perform action intent recognition processing on the target tracking data of the interaction object, and obtain the action intent information of the interaction object;

[0045] The game interaction control module is used to map the action intent information to the game interaction control information of the in-vehicle game corresponding to the game interaction request, so as to control the game character in the in-vehicle game to perform the corresponding action through the game interaction control information.

[0046] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0047] In response to a game interaction request triggered on the vehicle's in-vehicle platform, target detection processing is performed on the received video frame to obtain the target detection result of the video frame; the video frame is obtained by capturing the game interaction area of ​​the vehicle through the vehicle's camera device;

[0048] If the target detection result indicates that the video frame contains multiple candidate objects, then pose recognition processing is performed on the target tracking data of each candidate object to obtain the pose recognition result of each candidate object; the target tracking data is obtained by performing target tracking processing on the video frame;

[0049] If a target posture recognition result matching a preset posture is detected in the posture recognition results, the candidate object corresponding to the target posture recognition result is set as an interaction object, and the target tracking data of the interaction object is processed for action intent recognition to obtain the action intent information of the interaction object.

[0050] The action intent information is mapped to the game interaction control information of the in-vehicle game corresponding to the game interaction request, so as to control the game character in the in-vehicle game to perform the corresponding action through the game interaction control information.

[0051] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0052] In response to a game interaction request triggered on the vehicle's in-vehicle platform, target detection processing is performed on the received video frame to obtain the target detection result of the video frame; the video frame is obtained by capturing the game interaction area of ​​the vehicle through the vehicle's camera device;

[0053] If the target detection result indicates that the video frame contains multiple candidate objects, then pose recognition processing is performed on the target tracking data of each candidate object to obtain the pose recognition result of each candidate object; the target tracking data is obtained by performing target tracking processing on the video frame;

[0054] If a target posture recognition result matching a preset posture is detected in the posture recognition results, the candidate object corresponding to the target posture recognition result is set as an interaction object, and the target tracking data of the interaction object is processed for action intent recognition to obtain the action intent information of the interaction object.

[0055] The action intent information is mapped to the game interaction control information of the in-vehicle game corresponding to the game interaction request, so as to control the game character in the in-vehicle game to perform the corresponding action through the game interaction control information.

[0056] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0057] In response to a game interaction request triggered on the vehicle's in-vehicle platform, target detection processing is performed on the received video frame to obtain the target detection result of the video frame; the video frame is obtained by capturing the game interaction area of ​​the vehicle through the vehicle's camera device;

[0058] If the target detection result indicates that the video frame contains multiple candidate objects, then pose recognition processing is performed on the target tracking data of each candidate object to obtain the pose recognition result of each candidate object; the target tracking data is obtained by performing target tracking processing on the video frame;

[0059] If a target posture recognition result matching a preset posture is detected in the posture recognition results, the candidate object corresponding to the target posture recognition result is set as an interaction object, and the target tracking data of the interaction object is processed for action intent recognition to obtain the action intent information of the interaction object.

[0060] The action intent information is mapped to the game interaction control information of the in-vehicle game corresponding to the game interaction request, so as to control the game character in the in-vehicle game to perform the corresponding action through the game interaction control information.

[0061] The aforementioned game interaction control method, device, computer equipment, storage medium, and computer program product, in response to a game interaction request triggered on a vehicle's in-vehicle platform, perform target detection processing on the received video frame to obtain the target detection result of the video frame; the video frame is obtained by capturing the game interaction area of ​​the vehicle using the vehicle's camera; if the target detection result indicates that the video frame contains multiple candidate objects, then the target tracking data of each candidate object is processed for pose recognition to obtain the pose recognition result of each candidate object; the target tracking data is obtained by performing target tracking processing on the video frame; if a target pose recognition result matching a preset pose is detected in the pose recognition result, then the candidate object corresponding to the target pose recognition result is set as an interaction object, and the target tracking data of the interaction object is processed for action intent recognition to obtain the action intent information of the interaction object; then the action intent information is mapped to the game interaction control information of the in-vehicle game corresponding to the game interaction request, so as to control the game character in the in-vehicle game to perform the corresponding action through the game interaction control information. This method, after triggering a game interaction request on the in-vehicle platform, uses the vehicle's camera to capture video frames of the game interaction area. By performing target detection on the video frames, candidate objects can be detected, and it can be determined whether the candidate objects have triggered preset gestures. Candidate objects that have triggered preset gestures are set as target objects for subsequent in-vehicle game interaction control. By recognizing the action intent information of the interactive objects, the corresponding game interaction control is completed. This constructs a closed-loop interaction of "vehicle camera → target detection and tracking → action intent recognition → game interaction control", realizing a linkage mechanism of "perception-recognition-feedback". It can automatically determine whether the game interaction control is satisfied based on the video frames, greatly improving the effect and accuracy of controlling in-vehicle games through the actions of interactive objects. Attached Figure Description

[0062] Figure 1 This is a flowchart illustrating a game interaction control method in one embodiment;

[0063] Figure 2 This is a schematic diagram illustrating the application environment of a game interaction control method in one embodiment;

[0064] Figure 3 This is a flowchart illustrating the steps of performing target detection processing on a received video frame in one embodiment;

[0065] Figure 4 This is a schematic diagram of the target detection process in one embodiment;

[0066] Figure 5 This is a flowchart illustrating the action intent recognition process in one embodiment;

[0067] Figure 6This is a flowchart illustrating the game interaction control method in another embodiment;

[0068] Figure 7 This is a flowchart illustrating the game interaction control method in yet another embodiment;

[0069] Figure 8 This is a structural block diagram of a game interaction control device in one embodiment;

[0070] Figure 9 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0072] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0073] In one embodiment, such as Figure 1 As shown, a game interaction control method is provided. This embodiment illustrates the method by applying it to a computer device. It is understood that this method can also be applied to a server, and further to a system including a terminal (such as an in-vehicle terminal) and a server, and is implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0074] Step S101: In response to the game interaction request triggered on the vehicle's in-vehicle platform, target detection processing is performed on the received video frame to obtain the target detection result of the video frame; the video frame is obtained by capturing the game interaction area of ​​the vehicle through the vehicle's camera.

[0075] In this context, an in-vehicle platform refers to an integrated hardware and software infrastructure designed specifically for vehicles to support various vehicle functions. These functions include driving control, infotainment (such as in-vehicle games), network communication, and autonomous driving.

[0076] Among them, game interaction requests refer to interaction requests triggered by in-vehicle games on the in-vehicle platform.

[0077] The term "shooting equipment" refers to a device installed on a vehicle. For example, a shooting device could be a vehicle-mounted camera. It should be noted that the shooting device's field of view includes both inside and outside the vehicle.

[0078] The game interaction area refers to a pre-defined interaction recognition area for in-vehicle games. For example, to control an in-vehicle game through actions, the interacting object (such as a human body) needs to stand within the game interaction area for it to be recognized. Actions outside the game interaction area will not be recognized as game interaction control information. In practical applications, the game interaction area can be the front or rear of the vehicle, etc.

[0079] Figure 2 This is a schematic diagram illustrating the application environment of game interaction control methods. Specifically, such as... Figure 2 As shown, one or more in-vehicle games can be pre-installed on the vehicle's in-vehicle platform. Users can select the desired game via the platform's display screen, triggering a game interaction request. Alternatively, users can trigger the game interaction request via voice commands, such as saying, "Open XXX in-vehicle game." The in-vehicle platform then sends the game interaction request to a computer device. Responding to the user's game interaction request, the computer device controls the vehicle's camera to begin real-time filming of the game interaction area and uploads the captured video frames to the computer device, thus obtaining a continuous stream of video frames. The computer device can then use an object detection model to perform object detection processing on each video frame for preset candidate objects (such as the human body), outputting the bounding box information of the candidate objects in each frame to determine whether candidate objects appear in the game interaction area, and then performing subsequent game interaction control processing based on these candidate objects.

[0080] Furthermore, if the bounding box information is not null, it indicates that the video frame contains candidate objects. The computer device can then perform human keypoint recognition processing on the candidate objects in the video frame based on the bounding box information. For example, the local image corresponding to the bounding box information in the video frame can be cropped, and a keypoint detection model can be trained based on a deep learning-based object detection algorithm (such as the YOLO algorithm). Then, the keypoint detection model can be used to detect human keypoints in the local image. The keypoint detection model can first detect the human body region in the local image, and then detect the human keypoints within the human body region, thereby obtaining the keypoint information of the candidate objects (such as keypoint type and keypoint location). The computer device sets the bounding box information and keypoint information as the object detection result for that video frame.

[0081] Step S102: If the target detection result indicates that the video frame contains multiple candidate objects, then the target tracking data of each candidate object is processed for pose recognition to obtain the pose recognition result of each candidate object; the target tracking data is obtained by processing the video frame for target tracking.

[0082] The target tracking data includes the identification information of the tracked object (such as an interactive object or a candidate object) and the target detection results.

[0083] Specifically, if the target detection result indicates that a video frame contains a candidate object, then that candidate object can be designated as the interactive object. Based on the target detection result, a target tracking algorithm is used to perform target tracking processing on consecutive video frames for the interactive object. This continuously tracks the bounding box information and key point information of the interactive object in consecutive video frames and assigns unique identification information (such as ID information) to the interactive object. For example, the target tracking algorithm uses data association between the target detection result and the historical trajectory of the candidate object; it matches the bounding box information of adjacent video frames and calculates the matching score by combining the similarity between the key point information of adjacent video frames, and assigns unique ID information to the interactive object. Furthermore, it can predict the motion trajectory of the interactive object to handle occlusion or temporary loss of the interactive object. Finally, it outputs continuous bounding box information, key point information, and the motion trajectory of the interactive object carrying ID information, achieving stable tracking of the interactive object and ensuring that the interactive object maintains a consistent identity across consecutive video frames. The target detection results and identification information continuously detected for each video frame during the tracking process are output as the target tracking data for the interactive object.

[0084] If the target detection result indicates that the video frame contains multiple candidate objects, then based on the target detection result, a multi-target tracking algorithm is used to perform multi-target tracking processing on consecutive video frames to continuously track the bounding box information and key point information of each candidate object in consecutive video frames, and assign unique identification information (such as ID information) to each candidate object. For example, the target tracking algorithm is used to associate the target detection result with the historical trajectory of the candidate object; the bounding box information of adjacent video frames is matched, and the similarity between the key point information of adjacent video frames is combined to calculate the matching score, and a unique ID information is assigned to each candidate object. The motion trajectory of each candidate object can also be predicted to handle the occlusion or temporary loss of each candidate object. Finally, the continuous bounding box information, key point information, and motion trajectory of each candidate object carrying ID information are output, realizing stable tracking of each candidate object, thereby ensuring that each candidate object maintains the consistency of its identity between consecutive video frames, and outputting the target tracking data of each candidate object. For example, ByteTrack (a multi-target tracking algorithm) can be combined to achieve robust ID information tracking of multiple people targets, solving the problems of rapid movement, occlusion, and ID information drift of people in complex vehicle scenarios. Then, based on the bounding box information and key point information in the target detection results, the computer equipment performs pose recognition processing on the target tracking data of each candidate object according to the preset pose, and obtains the pose recognition result of each candidate object.

[0085] The pose recognition result is a "yes" or "no" classification result, used to characterize whether the candidate object has made a preset pose in the video frame. For example, the pose recognition result can be represented by "0 (no preset pose)" and "1 (preset pose made)".

[0086] Step S103: If a target pose recognition result matching the preset pose is detected in the pose recognition result, the candidate object corresponding to the target pose recognition result is set as the interactive object, and the action intention recognition processing is performed on the target tracking data of the interactive object to obtain the action intention information of the interactive object.

[0087] The preset posture refers to a posture that is pre-set for judging the target object. For example, the preset posture can be set to a raised hand posture.

[0088] Action intent information refers to the specific intention expressed by an interactive object when performing an action or behavior. For example, action intent information could be "jump," "squat," "lean to the left," or "lean to the right."

[0089] Specifically, the computer device sets the posture recognition result that matches the preset posture as the target posture recognition result, and then sets the candidate object corresponding to the target posture recognition result as the interactive object. For example, it sets the candidate object with a detected hand-raising posture as the interactive object, and then locks the identification information of the interactive object. In this game, the key point information in the target tracking data of the interactive object is processed for action intent recognition to obtain the action intent information of the interactive object.

[0090] Step S104: Map the action intent information to the game interaction control information of the in-vehicle game corresponding to the game interaction request, so as to control the game character in the in-vehicle game to perform the corresponding action through the game interaction control information.

[0091] Game interaction control information refers to computer program instructions used to control in-vehicle game interactions through the action intent information of interactive objects. Game interaction control information includes action control instructions for game characters, music control instructions for in-vehicle games, etc.

[0092] Specifically, the identified action intent information is mapped to the game interaction control information of the in-vehicle game corresponding to the game interaction request; the game interaction control information is used to control the in-vehicle game, such as controlling the actions of the game character (e.g., jumping, crouching). The vehicle's headlights can also project the game execution status and feedback results in real time or load them onto the vehicle's projection screen, enhancing the interactive perception of the interactive object.

[0093] In the aforementioned game interaction control method, in response to a game interaction request triggered on the vehicle's in-vehicle platform, target detection processing is performed on the received video frame to obtain the target detection result of the video frame; the video frame is obtained by capturing the game interaction area of ​​the vehicle through the vehicle's camera; if the target detection result indicates that the video frame contains multiple candidate objects, then pose recognition processing is performed on the target tracking data of each candidate object to obtain the pose recognition result of each candidate object; the target tracking data is obtained by performing target tracking processing on the video frame; if a target pose recognition result matching a preset pose is detected in the pose recognition result, then the candidate object corresponding to the target pose recognition result is set as the interaction object, and action intent recognition processing is performed on the target tracking data of the interaction object to obtain the action intent information of the interaction object; based on the action intent information, game interaction control information corresponding to the game interaction request is obtained. This method, after triggering a game interaction request on the in-vehicle platform, uses the vehicle's camera to capture video frames of the game interaction area. By performing target detection on the video frames, candidate objects can be detected, and it can be determined whether the candidate objects have triggered preset gestures. Candidate objects that have triggered preset gestures are set as target objects for subsequent in-vehicle game interaction control. By recognizing the action intent information of the interactive objects, the corresponding game interaction control is completed. This constructs a closed-loop interaction of "vehicle camera → target detection and tracking → action intent recognition → game interaction control", realizing a linkage mechanism of "perception-recognition-feedback". It can automatically determine whether the game interaction control is satisfied based on the video frames, greatly improving the effect and accuracy of controlling in-vehicle games through the actions of interactive objects.

[0094] In one embodiment, such as Figure 3 As shown, step S101 above performs target detection processing on the received video frame to obtain the target detection result of the video frame, specifically including the following:

[0095] Step S301: Obtain the brightness information of the target area in the video frame.

[0096] The target region refers to the selected area within a video frame. For example, the target region can be a local area in the lower middle part of the video frame (such as 3 / 5). The selection of the target region can be obtained from experimental testing, set based on empirical values, or flexibly set according to the needs of the actual application scenario.

[0097] Among them, brightness information refers to information describing the brightness of the target area in a video frame.

[0098] Figure 4This is a flowchart illustrating the target detection process. After receiving continuous video frames captured in real-time by the vehicle's camera, the computer device can first check whether the lighting conditions of the video frames meet preset lighting conditions. This determines whether to proceed to subsequent steps such as target detection and pose recognition. For example, if the lighting conditions are met, the video frames are input into the target detection model, and the bounding box information and key point information of the candidate objects are output. If the lighting conditions are not met, target detection processing is not performed. Specifically, the terminal calculates the brightness information of the target region in the video frame. This can be done by calculating the brightness histogram of the lower 3 / 5 region of the video frame and using this calculated brightness histogram as the brightness information.

[0099] Step S302: Based on the brightness information, obtain the mean brightness and skewness coefficient of the target area.

[0100] The average brightness value refers to the average brightness value of the target area. For example, the average brightness value could be the average brightness value of all pixels in the target area.

[0101] The skewness coefficient is used to reflect the degree of skewness in the brightness distribution of the target area.

[0102] Specifically, the computer device calculates the mean luminance and skewness coefficient of the target region in the video frame based on the luminance histogram. For example, the luminance histogram includes the luminance value corresponding to each pixel and the frequency of each luminance value. Based on the luminance values ​​of all pixels in the target region, the average luminance of a single pixel in the target region can be calculated, thus obtaining the mean luminance of the target region. The computer device can also use the mean luminance and standard deviation of the target region to calculate the skewness coefficient of the target region.

[0103] Step S303: If the mean brightness and skewness coefficient both meet the preset illumination conditions, then target detection processing is performed on the video frame to obtain the target detection result of the video frame.

[0104] The preset lighting conditions refer to the pre-set judgment conditions used to analyze the lighting conditions of the target area in the video frame. For example, the preset lighting conditions could be whether the average brightness of the target area in the video frame reaches a first threshold and whether the skewness coefficient of the target area reaches a second threshold. In practical applications, the brightness value of an image is usually between 0 and 255. When the brightness value is less than 50, it is difficult for the human eye to recognize, and when the brightness value exceeds 200, it is overexposed. Therefore, if the target area in the video frame needs to have good lighting conditions, the first threshold corresponding to the average brightness can be set in the range of 100-150, for example, setting the first threshold to 120. Generally, when the skewness coefficient is in the range of (-0.5, 0.5), the brightness distribution of the image is close to symmetrical. When the skewness coefficient is greater than 1.0 or less than -1.0, the brightness on one side of the image is too bright (i.e., strong light), and the brightness on the other side is too dark (i.e., strong backlight). Therefore, if the target area in the video frame needs to have good lighting conditions, the second threshold corresponding to the skewness coefficient can be set in the range of (-0.9, 0.9).

[0105] Specifically, the computer device can set a first threshold for the average brightness and a second threshold for the skewness coefficient, and set lighting conditions based on the first and second thresholds. If the average brightness and skewness coefficient of the target area in the video frame both meet the lighting conditions, it indicates that the lighting conditions of the target area in the video frame are good, and the video frame will not be too dark or too bright. Then, the target detection processing of step S101 above will continue to be performed on the video frame. If at least one of the average brightness and / or skewness coefficient of the target area in the video frame does not meet the lighting conditions, it indicates that the lighting conditions of the target area in the video frame are mixed, and the video frame may be too dark (e.g., low illumination) or too bright (e.g., strong backlight). Then, it is not necessary to perform the target detection processing of step S101 above on the video frame, and the video frame acquisition of the vehicle's shooting device will continue to be maintained, and the above lighting condition judgment will continue to be performed on the next video frame.

[0106] Therefore, if the mean brightness and skewness coefficient of the video frame meet the lighting conditions, the subsequent processing flow (e.g., target detection + (multi) target tracking + pose recognition + action intent recognition) is executed; if the conditions are not met, the subsequent processing flow (e.g., target detection + (multi) target tracking + pose recognition + action intent recognition) is skipped, and only the video frame acquisition of the shooting device is maintained until the mean brightness and skewness coefficient of the video frame are detected to meet the lighting conditions.

[0107] In this embodiment, by judging the lighting conditions by the average brightness and skewness coefficient of the target area in the video frame, the judgment mechanism is equivalent to using this judgment mechanism as a pre-filter for subsequent processing. This avoids subsequent processing of video frames with low illumination or strong backlight, avoids invalid detection and analysis of video frames, and effectively improves the effectiveness and resource utilization of subsequent processing.

[0108] In one embodiment, step S303 above, if the mean brightness and skewness coefficient both meet the preset illumination conditions, then target detection processing is performed on the video frame to obtain the target detection result of the video frame. Specifically, this includes the following: using a target detection model, feature processing is performed on the video frame in both the channel dimension and the spatial dimension to obtain target feature information of the video frame; the target detection model is obtained by optimizing the feature processing module of the base model through a multi-branch attention mechanism; using the target detection model, target detection processing is performed on the video frame based on the target feature information to obtain bounding box information for candidate objects in the video frame; based on the bounding box information, key point recognition processing is performed on the candidate objects in the video frame to obtain key point information of the candidate objects; and the target detection result is obtained based on the bounding box information and the key point information.

[0109] Among them, the object detection model refers to the model used to detect whether there are candidate objects in a video frame and the position of the candidate objects in the video frame.

[0110] To improve the accuracy of target detection for candidate objects in video frames, this application also improves the base model used for target detection, resulting in an improved target detection model. This improved model enables accurate identification of candidate objects. Specifically, a computer device can construct a base model based on a target detection algorithm. Then, a multi-branch attention mechanism can be used to improve and optimize the feature processing module in the base model, enabling the module to process both channel and spatial features of the video frame. This allows for more accurate perception of key features (i.e., features related to candidate objects) and key regions (i.e., regions containing candidate objects) within the video frame, leading to the improved detection model. The detection model is then iteratively trained using video frame samples. After meeting preset training termination conditions (such as reaching a preset maximum number of training rounds or the accuracy of the model output no longer improving), the computer device obtains the trained target detection model.

[0111] The computer device inputs video frames into the object detection model. The improved feature processing module in the object detection model performs feature extraction processing on the video frames in both the channel dimension and the spatial dimension, obtaining feature data in the channel dimension and feature data in the spatial dimension. Then, the feature data in the channel dimension and feature data in the spatial dimension are weighted to obtain the target feature information of the video frame. Subsequently, the object detection model performs target detection processing on the video frame based on the target feature information, obtaining the bounding box information (BBox) of the candidate object in the video frame.

[0112] If the bounding box information is not empty, it indicates that a candidate object has been detected in the video frame. The computer device can then extract the image within the bounding box information to obtain the target image of the candidate object in the video frame. Human keypoint recognition processing is then performed on the target image to obtain the keypoint information of the candidate object in the target image. The computer device can then output both the bounding box information and the keypoint information as the object detection result.

[0113] For example, a base model can be built using the YOLO algorithm (You Only Look Once, a deep learning-based object detection algorithm), such as using the YOLO v11 algorithm. The neck network in the YOLO algorithm is a key part connecting the backbone network and the head, responsible for multi-scale feature fusion to improve the model's ability to detect objects of different sizes. The computer device adds a Convolutional Block Attention Module (CBAM) to the neck network of the Feature Pyramid Network (FPN) structure. The CBAM is designed as a lightweight structure with very few parameters, and its introduction has minimal impact on the model's inference speed, ultimately resulting in an improved YOLO model. The improved YOLO model is then trained to obtain a trained YOLO model. This trained YOLO model is then converted into a serialized ".engine" file format of the inference engine optimized by TensorRT (a deep learning inference optimization technique used to improve inference speed through layer fusion, accuracy calibration, and automatic kernel tuning) for efficient inference. The serialized ".engine" file format is a binary file containing the improved YOLO model's structure, parameters, and computation graph, which can be directly loaded into a Graphics Processing Unit (GPU) for execution, thus further improving the inference speed of the trained YOLO model (i.e., the object detection model).

[0114] In this embodiment, by introducing a multi-branch attention mechanism into the feature processing module of the base model, the improved and trained object detection model can more accurately perceive the features related to candidate objects and the regions containing candidate objects in the video frame. This reduces the influence of invalid features on the model's object detection processing, thereby greatly improving the accuracy and stability of the bounding box information output by the object detection model. It also provides a reliable processing basis for key point recognition, and further improves the accuracy of the key point information of candidate objects in the obtained video frame.

[0115] In one embodiment, step S103, which involves performing action intent recognition processing on the target tracking data of the interactive object to obtain action intent information of the interactive object, specifically includes the following: performing action intent recognition processing on key point information in the target tracking data of the interactive object for a first action to obtain first action intent information of the interactive object; if the representation of the first action intent information does not conform to the first action, then performing action intent recognition processing on key point information in the continuous multi-frame target tracking data for a second action to obtain second action intent information of the interactive object; if the representation of the second action intent information does not conform to the second action, then performing action intent recognition processing on key point information in the continuous multi-frame target tracking data for a third action to obtain third action intent information of the interactive object; wherein the first action, the second action, and the third action are different.

[0116] The first, second, and third actions refer to the physical movements performed in an interactive manner. For example, the first, second, and third actions could be squatting, tilting the body to the left or right, jumping, etc.

[0117] Action intent information is used to characterize whether the interactive object has performed a corresponding action. For example, first action intent information indicates whether the interactive object has performed a first action. Second action intent information indicates whether the interactive object has performed a second action. Third action intent information indicates whether the interactive object has performed a third action.

[0118] Specifically, after identifying the interaction object, the computer device can perform motion intent recognition processing on the interaction object for the first action (such as a squatting action, down) based on the bounding box information and target key point information associated with the first action in the target tracking data of the interaction object across multiple consecutive frames. This is to detect whether the interaction object has triggered the first action (i.e., whether the interaction object has made the posture of the first action) and output the first action intent information of the interaction object. If the first action intent information does not match the first action, it means that the interaction object has not triggered the first action. The computer device can then continue to detect whether the interaction object has triggered other actions. For example, the computer device can perform motion intent recognition processing on the interaction object for the second action (such as a left or right leaning action) based on the bounding box information and target key point information associated with the second action in the target tracking data of the interaction object across multiple consecutive frames. This is to detect whether the interaction object has triggered the second action (i.e., whether the interaction object has made the posture of the second action) and output the second action intent information of the interaction object. If the second action intent information representation does not match the second action, that is, the interactive object neither triggered the first action nor the second action, the computer device can continue to detect whether the interactive object has triggered other actions. For example, the computer device can perform action intent recognition processing for the third action (such as the jump action, up) on the interactive object based on the bounding box information and the target key point information associated with the third action in the target tracking data of the interactive object in consecutive multi-frames, so as to detect whether the interactive object has triggered the third action (that is, whether the interactive object has made the posture of the third action) and output the third action intent information of the interactive object.

[0119] It should be noted that, in addition to judging the first, second, and third actions, other actions can also be judged, such as the fourth, fifth, and sixth actions. Furthermore, the order of judgment between actions can be flexibly adjusted. For example, in practical applications, the third action can be judged first, then the first action, and finally the second action. However, the order of judgment between actions needs to be designed with priority logic; multiple actions cannot be judged simultaneously to avoid false triggering due to overlapping actions.

[0120] In this embodiment, based on the key point information and bounding box information in the target tracking data of the interactive object in multiple consecutive frames, the video frames of the interactive object in multiple consecutive frames are processed for action intent recognition for the first action, the second action, and the third action, respectively, to obtain the first action intent information, the second action intent information, and the third action intent information of the interactive object, which lays the foundation for controlling the actions of the game character in subsequent steps.

[0121] In one embodiment, the key point information of multiple consecutive frames is processed for action intent recognition of a first action to obtain the first action intent information of the interactive object. Specifically, this includes: obtaining the hip position information corresponding to the hip key point and the knee position information corresponding to the knee key point in the key point information; obtaining the height difference information between the hip key point and the knee key point based on the position difference between the hip position information and the knee position information; and determining whether the interactive object has triggered the first action based on the height difference information of multiple consecutive frames to obtain the first action intent information of the interactive object.

[0122] Specifically, hip position information refers to the pixel position of the interactive object's hip in the video frame. Knee position information refers to the pixel position of the interactive object's knee in the video frame. Pixel positions can be represented using X-axis and Y-axis coordinates, i.e., pixel position = (X, Y).

[0123] Specifically, assuming the first action is a squatting motion, the target keypoint information associated with the squatting motion includes the hip and knee of the human body. The computer device can then extract the hip position information (X1, Y1) corresponding to the hip keypoint and the knee position information (X2, Y2) corresponding to the knee keypoint from the keypoint information of the target tracking data of the interactive object across multiple consecutive frames. Based on the difference in the Y-axis coordinates between the hip and knee position information, i.e., Y1-Y2, the computer device calculates the height difference information between the hip and knee keypoints; it calculates the overall height of the interactive object based on the bounding box information; and then calculates the ratio between the height difference information and the overall height of the human body. If this ratio is less than or equal to a first ratio threshold, it indicates that the hip of the interactive object is close to or lower than the knee, indicating that the interactive object has triggered a squatting motion. To avoid misjudgment based on a single height difference, the computer device can determine whether the interactive object has triggered the first action based on the ratio between the height difference information in multiple consecutive video frames and the overall height of the human body. For example, if the ratio is less than or equal to a first ratio threshold for a first number of video frames (e.g., 3 consecutive frames), the output will show the first action intent information indicating that the interactive object has triggered the first action; otherwise, the output will show the first action intent information indicating that the interactive object has not triggered the first action.

[0124] The first proportional threshold can be adjusted through pose normalization and adaptive sensitivity strategies to improve adaptability to different users. Pose normalization and adaptive sensitivity strategy adjustment is a technique that combines computer vision (such as human pose recognition) and dynamic parameter optimization. For example, after keypoint information is identified, normalization is used to scale all keypoint information to a uniform proportion. Then, an adaptive sensitivity strategy is used to dynamically calculate the proportional threshold for the current frame by combining the normalized keypoint information from multiple consecutive frames.

[0125] In this embodiment, the height difference information between the hip key point and the knee key point, as well as the overall height of the human body, is used to calculate the ratio between the height difference information and the overall height of the human body. Then, the trend of the ratio value change over multiple consecutive frames is used to determine whether the interactive object has triggered the first action, thus achieving accurate judgment of the intention of the first action. Moreover, by utilizing multi-frame information, the temporal sequence of video frames can be used to combat the problems of slight noise and pose blur in video frames, effectively improving the accuracy of the intention of the first action.

[0126] In one embodiment, the key point information of multiple consecutive frames is processed for intent recognition of a third action to obtain the third action intent information of the interactive object. Specifically, this includes: obtaining the ankle position information corresponding to the ankle key point in the key point information; obtaining the knee displacement information of the knee key point based on the knee position information of multiple consecutive frames; obtaining the ankle displacement information of the ankle key point based on the ankle position information of multiple consecutive frames; and determining whether the interactive object has triggered a third action based on the knee displacement information and the ankle displacement information to obtain the third action intent information of the interactive object.

[0127] Among them, ankle position information refers to the pixel position of the ankle (foot) of the interactive object in the video frame.

[0128] Specifically, assuming the third action is a jump, the target keypoint information associated with the jump includes the knee and ankle of the human body. Furthermore, the computer device can extract the ankle position information corresponding to the ankle keypoint from the keypoint information of the target tracking data of the interactive object across multiple consecutive frames. The computer device can construct a "stack" structure class to store the ankle and knee position information of the interactive object in a second consecutive number of video frames (e.g., 5 consecutive frames), and update it in a timely manner. By calculating the Y-axis displacement value of the ankle keypoint in the current frame and the first frame in the "stack" structure, the ankle displacement information Δy1 is obtained; and by calculating the Y-axis displacement value of the knee keypoint in the current frame and the first frame in the "stack" structure, the knee displacement information Δy2 is obtained. Determine whether the ankle displacement information Δy1 is greater than a preset ankle displacement threshold, and determine whether the knee displacement information Δy2 is greater than a preset knee displacement threshold. If both the ankle displacement information Δy1 and the knee displacement information Δy2 are greater than the preset knee displacement threshold, it indicates that the interactive object has triggered a jump action, and outputs the third action intent information indicating that the interactive object has triggered a third action. Otherwise, outputs the third action intent information indicating that the interactive object has not triggered a third action.

[0129] Among them, the knee displacement threshold and ankle displacement threshold can be adjusted through posture normalization and adaptive sensitivity strategies to improve adaptability to different users.

[0130] In this embodiment, by comparing ankle and knee displacement information from multiple consecutive frames with a displacement threshold, it is possible to determine whether the interactive object has undergone significant displacement, thereby accurately determining whether the interactive object has triggered a third action. During the action intent recognition process, different actions are judged sequentially according to priority logic, effectively preventing errors in action intent recognition caused by pre-gesture shaking or multiple overlapping actions, improving the accuracy of third action intent information recognition, and thus enhancing the logical rigor of the interaction and the consistency of the user experience.

[0131] In one embodiment, the key point information of multiple consecutive frames is processed for intent recognition of the second action to obtain the second action intent information of the interactive object. Specifically, this includes: obtaining the left shoulder position information corresponding to the left shoulder key point and the right shoulder position information corresponding to the right shoulder key point in the key point information; obtaining the shoulder line of the interactive object based on the left shoulder position information and the right shoulder position information; determining the angle between the shoulder line and the target bounding box line in the bounding box information of the target tracking data of the interactive object; and determining whether the interactive object has triggered the second action based on the angle of multiple consecutive frames to obtain the second action intent of the interactive object.

[0132] The left-click position information refers to the pixel position of the left shoulder of the interactive object in the video frame. The right-shoulder position information refers to the pixel position of the right shoulder of the interactive object in the video frame.

[0133] Specifically, assuming the second action is a leftward or rightward tilt, the target keypoint information associated with the leftward or rightward tilt includes the shoulder keypoints of the interactive object. The computer device selects the shoulder keypoints of the interactive object to form a line, that is, based on the left and right shoulder position information, connects the left and right shoulder keypoints to obtain the shoulder line of the interactive object. The target bounding box line in the bounding box information of the interactive object is determined, for example, the target bounding box line is the bottom bounding box line. Then, the shoulder line forms an angle with the target bounding box line in the bounding box information. For example, the shoulder line and the target bounding box line can be extended to intersect and form an angle, and then the angle between the shoulder line and the target bounding box line is calculated. If the absolute value of the angle is greater than a preset angle threshold (e.g., 15 degrees), it indicates that the interactive object has triggered a tilt action. Further, the tilt direction is combined to determine whether the tilt action is "left" or "right," and finally, the second action intent information indicating that the interactive object has triggered the second action (e.g., tilting left or right) is output; otherwise, the second action intent information indicating that the interactive object has not triggered the second action is output.

[0134] Among them, the preset angle threshold can be adjusted through posture normalization and adaptive sensitivity strategies to improve the adaptability to different users.

[0135] In this embodiment, the angle between the shoulder line connecting the key points of the interactive object's shoulders and the target bounding box line in the bounding box information is formed by the two lines. Comparing this angle with a preset angle threshold allows for the determination of whether the interactive object's body is tilted, thus accurately determining whether the interactive object has triggered a second action. Furthermore, the tilt direction of the second action is determined by combining the angle direction. During the action intent recognition process, different actions are judged sequentially according to priority logic, effectively preventing errors in action intent recognition caused by forward swaying or multiple overlapping actions. This improves the accuracy of recognizing the second action intent information, thereby enhancing the logical rigor of the interaction and the consistency of the user experience.

[0136] In one embodiment, step S102 above, which performs pose recognition processing on the target tracking data of each candidate object to obtain the pose recognition result of each candidate object, specifically includes the following: extracting the bounding box region of the candidate object from the target tracking data of each candidate object; performing pose classification processing on the bounding box region of each candidate object by using a pose classifier for a preset pose to obtain the pose recognition result of each candidate object; wherein, the pose recognition result is used to characterize whether the pose of the candidate object matches the preset pose.

[0137] A pose classifier is a machine learning model used to classify input data (such as bounding box regions) into predefined discrete categories (e.g., belonging to a preset pose category or not belonging to a preset pose category). In practical applications, lightweight models can be used to build pose classifiers.

[0138] A pose classifier for a preset pose (such as a raised hand) is pre-trained. Specifically, a lightweight detection network (model size 1.3M-7M) of DBFace (Dual-Branch Face Detection Network, an efficient and lightweight neural network for face detection) can be reconstructed as a pose classifier for the raised hand pose. This classifier is trained using a dataset of a large number of labeled raised hand pose samples, achieving a 96.8% pose recognition accuracy while maintaining its original lightweight advantage. The trained pose classifier is then converted into a TensorRT-optimized ".engine" inference engine, leveraging TensorRT's layer fusion and accuracy calibration techniques to comprehensively improve the inference efficiency of the pose classifier.

[0139] Furthermore, based on the bounding box information in the target tracking data of each video frame of the candidate object, the bounding box region corresponding to the candidate object is extracted from each video frame; the bounding box region is input into the pose classifier, and the pose classifier performs pose classification processing on the bounding box region for the preset pose, and outputs the pose recognition result of the candidate object, so as to determine whether the candidate object has made the preset pose (such as raising a hand) based on the pose recognition result.

[0140] In practical applications, computer devices perform secondary pose classification on the bounding box regions of candidate objects in video frames. If a candidate object is detected raising its hand, it indicates that the candidate object is an object actively participating in the in-vehicle game. The identification information (such as ID information) of the candidate object is then locked as the main object of game interaction (i.e., the interaction object). Only the action behavior of the interaction object is tracked and identified to improve the discrimination efficiency and effectively filter irrelevant interference actions. This can effectively solve the target competition problem in multi-person collaborative interaction scenarios.

[0141] In this embodiment, the bounding box region of each candidate object is extracted from the target tracking data of each candidate object. Then, a pose classifier for the preset pose is used to classify the bounding box region of each candidate object to obtain the pose recognition result of each candidate object. The pose recognition result is used to determine whether a candidate object has triggered the preset pose, thereby realizing the locking and tracking of interactive objects in the in-vehicle game. By using a lightweight pose recognizer to classify and detect the preset pose, the classification accuracy is ensured while the classification efficiency is effectively improved.

[0142] In one embodiment, game interaction control information includes action control instructions for game characters.

[0143] Step S104 above maps the action intent information to the game interaction control information of the in-vehicle game corresponding to the game interaction request, so as to control the game character in the in-vehicle game to perform the corresponding action through the game interaction control information. Specifically, it includes the following: determining the game character in the in-vehicle game corresponding to the game interaction request; mapping the action intent information to the action control instructions of the game character, so as to control the game character to perform the corresponding action through the action control instructions.

[0144] Among them, motion control instructions refer to computer program instructions that control game characters to perform game actions (such as crouching, jumping, tilting, turning, etc.).

[0145] Specifically, Figure 5 This diagram illustrates the process of motion intent recognition and game interaction control. The computer device utilizes keypoint and bounding box information from multiple consecutive frames of the interactive object to perform motion intent recognition for actions such as crouching, leaning left / right, and jumping. After motion intent recognition is complete, the computer device uses a "field update unit" to immediately write the current motion intent information into a state cache. The "motion mapping unit" then translates this information into action control commands (e.g., up, down, left / right) corresponding to the game character in the in-vehicle game, allowing the game character to perform actions like jumping, crouching, or leaning left / right. It's important to note that the computer device employs a predefined command priority strategy in the motion intent recognition process: once an action (e.g., crouching) meets the judgment criteria, the computer device immediately uses the "field update unit" to write the corresponding motion intent information and terminates the subsequent motion intent recognition process (e.g., leaning or jumping judgment) for the current video frame. This effectively avoids misjudgments caused by continuous actions or pre-motion animations, improving recognition accuracy and response efficiency.

[0146] Upon receiving motion control commands, the computer device uses a "game command mapping unit" to precisely map these commands to specific motion controls of the game character, enabling the character to perform corresponding actions such as jumping, crouching, and tilting. This is achieved through a standardized interface protocol, enabling low-latency data communication with the in-vehicle game engine, ensuring that motion intent information seamlessly and efficiently drives the in-vehicle game's response. Furthermore, the game's execution status and feedback results can be projected in real-time or loaded onto the vehicle's projection screen, enhancing the interactive experience.

[0147] In this embodiment, by mapping the action intent information of the interactive object to the action control instructions of the game character in the in-vehicle game corresponding to the game interaction request, the game character can be controlled to perform corresponding actions through the action control instructions. This greatly enriches the interactive effect of the in-vehicle game, enhances the entertainment value of the in-vehicle game, and has broad application prospects.

[0148] In one embodiment, such as Figure 6 As shown, another method for game interaction control is provided. Taking the application of this method to a computer device as an example, it includes the following steps:

[0149] Step S601: In response to a game interaction request triggered on the vehicle's in-vehicle platform, obtain the brightness information of the target area in the video frame; the video frame is obtained by capturing the game interaction area of ​​the vehicle using the vehicle's camera.

[0150] Step S602: Based on the brightness information, obtain the mean brightness and skewness coefficient of the target area.

[0151] Step S603: If the mean brightness and skewness coefficient both meet the preset lighting conditions, then the target detection model performs feature processing on the video frame in both the channel dimension and the spatial dimension to obtain the target feature information of the video frame; the target detection model is obtained by optimizing the feature processing module of the base model through a multi-branch attention mechanism.

[0152] Step S604: Using the target detection model, target detection processing is performed on the video frame based on the target feature information to obtain the bounding box information of the candidate object in the video frame.

[0153] Step S605: Based on the bounding box information, perform key point recognition processing on the candidate objects in the video frame to obtain the key point information of the candidate objects.

[0154] Step S606: Obtain the target detection result based on the bounding box information and key point information.

[0155] Step S607: If the target detection result indicates that the video frame contains multiple candidate objects, then the target tracking data of each candidate object is processed for pose recognition to obtain the pose recognition result of each candidate object; the target tracking data is obtained by performing target tracking processing on the video frame.

[0156] Step S608: If a target pose recognition result matching the preset pose is detected in the pose recognition result, the candidate object corresponding to the target pose recognition result is set as the interactive object, and the action intention recognition processing is performed on the target tracking data of the interactive object to obtain the action intention information of the interactive object.

[0157] Step S609: Map the action intent information to the game interaction control information of the in-vehicle game corresponding to the game interaction request, so as to control the game character in the in-vehicle game to perform the corresponding action through the game interaction control information.

[0158] The aforementioned game interaction control method can achieve the following beneficial effects: After the in-vehicle platform triggers a game interaction request, video frames of the game interaction area of ​​the vehicle are captured using the vehicle's camera. By performing target detection on the video frames, candidate objects in the video frames can be detected, and it can be identified whether the candidate objects have triggered preset postures. Candidate objects that have triggered preset postures are set as target objects for subsequent in-vehicle game interaction control. By recognizing the action intent information of the interaction object, the corresponding game interaction control is completed. A closed-loop interaction of "vehicle camera → target detection and tracking → action intent recognition → game interaction control" is constructed, realizing a linkage mechanism of "perception-recognition-feedback". It can automatically determine whether the game interaction control is satisfied based on the video frames, greatly improving the accuracy of controlling in-vehicle games through the actions of interaction objects.

[0159] To more clearly illustrate the game interaction control method provided in this disclosure, a specific embodiment will be used to describe the above-mentioned game interaction control method in detail below. For example... Figure 7 As shown, another method for game interaction control is provided, which can be applied to computer devices, and specifically includes the following:

[0160] (1) Video acquisition module: acquires video frames in real time from the vehicle camera and provides continuous video frames as video input data.

[0161] (2) Human body detection module: Performs target detection on the acquired video frames and outputs basic visual information such as the bounding box information, key point category and key point position of the human body (i.e., candidate object).

[0162] (3) Human body tracking module: Based on the bounding box information of the detected human body, a multi-target tracking algorithm is used to assign a unique ID information to each human body and continuously track it to maintain the consistency of each human body's identity across multiple video frames.

[0163] (4) Hand-raising posture recognition module: Based on the bounding box information of the human body, the human body image area is cropped from the video frame; the human body image area is used to identify whether the human body has a hand-raising action. If the human body has a hand-raising action, the ID information of the human body is locked as the subsequent interaction object.

[0164] (5) Human motion intention recognition module: Analyze the key point category and key point position of the locked interactive object, identify the specific motion intention of the interactive object, such as jumping, squatting, leaning left, leaning right, etc., and output motion intention information.

[0165] (6) Game control information mapping module: The identified action intention information is mapped into action control information to realize the action control of the game character in the in-vehicle game (such as jumping and crouching) and complete the natural human-computer interaction experience.

[0166] In this embodiment, a complete technology chain from visual perception and posture recognition to intent analysis and in-vehicle game control is established. It can map the recognized action intent information into action control commands in real time, realizing natural and efficient human-computer interactive game control operation. This not only improves the accuracy of controlling in-vehicle games through the actions of interactive objects, but also enhances the interaction effect between interactive objects and in-vehicle games.

[0167] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0168] Based on the same inventive concept, this application also provides a game interaction control device for implementing the game interaction control method described above. The solution provided by this device is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more game interaction control device embodiments provided below can be found in the limitations of the game interaction control method described above, and will not be repeated here.

[0169] In one embodiment, such as Figure 8 As shown, a game interaction control device 800 is provided, including: a target detection module 801, a posture recognition module 802, an action intent recognition module 803, and a game interaction control module 804, wherein:

[0170] The target detection module 801 is used to respond to the game interaction request triggered on the vehicle's in-vehicle platform, perform target detection processing on the received video frame, and obtain the target detection result of the video frame; the video frame is obtained by capturing the game interaction area of ​​the vehicle through the vehicle's camera.

[0171] The pose recognition module 802 is used to perform pose recognition processing on the target tracking data of each candidate object if the target detection result represents a video frame containing multiple candidate objects, so as to obtain the pose recognition result of each candidate object; the target tracking data is obtained by performing target tracking processing on the video frame.

[0172] The action intent recognition module 803 is used to set the candidate object corresponding to the target pose recognition result as the interaction object if the pose recognition result is detected to match the preset pose, and to perform action intent recognition processing on the target tracking data of the interaction object to obtain the action intent information of the interaction object.

[0173] The game interaction control module 804 is used to map action intent information to game interaction control information of the in-vehicle game corresponding to the game interaction request, so as to control the game character in the in-vehicle game to perform corresponding actions through the game interaction control information.

[0174] In one embodiment, the target detection module 801 is further configured to acquire brightness information of the target region in the video frame; based on the brightness information, obtain the mean brightness and skewness coefficient of the target region; if the mean brightness and skewness coefficient both meet the preset illumination conditions, then perform target detection processing on the video frame to obtain the target detection result of the video frame.

[0175] In one embodiment, the object detection module 801 is further configured to perform feature processing on the video frame in both the channel dimension and the spatial dimension using an object detection model to obtain target feature information of the video frame; the object detection model is obtained by optimizing the feature processing module of the base model through a multi-branch attention mechanism; based on the target feature information, the object detection model performs object detection processing on the video frame to obtain bounding box information of candidate objects in the video frame; based on the bounding box information, key point recognition processing is performed on the candidate objects in the video frame to obtain key point information of the candidate objects; and based on the bounding box information and key point information, the object detection result is obtained.

[0176] In one embodiment, the action intent recognition module 803 is further configured to perform action intent recognition processing on key point information in target tracking data of multiple consecutive frames of the interactive object for a first action, to obtain first action intent information of the interactive object; if it is detected that the representation of the first action intent information does not conform to the first action, then perform action intent recognition processing on key point information of multiple consecutive frames for a second action, to obtain second action intent information of the interactive object; if it is detected that the representation of the second action intent information does not conform to the second action, then perform action intent recognition processing on key point information of multiple consecutive frames for a third action, to obtain third action intent information of the interactive object; wherein, the first action, the second action, and the third action are different.

[0177] In one embodiment, the game interaction control device 800 further includes a first action recognition module, used to acquire hip position information corresponding to the hip key point and knee position information corresponding to the knee key point in the key point information; obtain height difference information between the hip key point and the knee key point based on the position difference between the hip position information and the knee position information; and determine whether the interactive object triggers a first action based on the height difference information of multiple consecutive frames to obtain the first action intention information of the interactive object.

[0178] In one embodiment, the game interaction control device 800 further includes a third action recognition module, used to obtain ankle position information corresponding to the ankle key point in the key point information; obtain knee displacement information of the knee key point based on knee position information of multiple consecutive frames; obtain ankle displacement information of the ankle key point based on ankle position information of multiple consecutive frames; and determine whether the interactive object triggers a third action based on the knee displacement information and ankle displacement information, thereby obtaining the third action intent information of the interactive object.

[0179] In one embodiment, the game interaction control device 800 further includes a second action recognition module, used to acquire left shoulder position information corresponding to the left shoulder key point and right shoulder position information corresponding to the right shoulder key point in the key point information; obtain the shoulder line of the interactive object based on the left shoulder position information and the right shoulder position information; determine the angle between the shoulder line and the target frame line in the bounding box information of the target tracking data of the interactive object; and determine whether the interactive object triggers a second action based on the angle of multiple consecutive frames, thereby obtaining the second action intention of the interactive object.

[0180] In one embodiment, the pose recognition module 802 is further configured to extract the bounding box region of each candidate object from the target tracking data of each candidate object; and to perform pose classification processing on the bounding box region of each candidate object by using a pose classifier for a preset pose to obtain the pose recognition result of each candidate object; wherein the pose recognition result is used to characterize whether the pose of the candidate object matches the preset pose.

[0181] Each module in the aforementioned game interaction control device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0182] In one embodiment, a computer device is provided, which may be a vehicle-mounted terminal, and its internal structure diagram may be as follows: Figure 9As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a game interaction control method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0183] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0184] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0185] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0186] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0187] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0188] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0189] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A game interaction control method, characterized by, The method comprises: In response to a game interaction request triggered by an in-vehicle platform of a vehicle, feature processing of a received video frame is performed in a channel dimension and a spatial dimension respectively by an improved feature processing module in a target detection model, to obtain feature data of the video frame in the channel dimension and feature data of the video frame in the spatial dimension, and the feature data in the channel dimension and the feature data in the spatial dimension are subjected to weighted processing to obtain target feature information of the video frame; wherein the improved feature processing module is obtained by adding a lightweight multi-branch attention mechanism to a feature processing module of a base model; Based on the target feature information, target detection processing is performed on the video frame to obtain a target detection result of the video frame; the video frame is obtained by photographing a game interaction area of the vehicle by a photographing device of the vehicle; the game interaction area is located outside the vehicle; If the target detection result indicates that a plurality of candidate objects are included in the video frame, a posture classifier for a preset posture is used to perform binary classification processing on target tracking data of each candidate object for the preset posture, to obtain a posture recognition result of each candidate object; the target tracking data is obtained by performing target tracking processing on the video frame; the posture recognition result is used to indicate whether the posture of the candidate object matches the preset posture; the posture classifier for the preset posture is obtained based on a face detection model; If a target posture recognition result that matches the preset posture is detected in the posture recognition result, the candidate object corresponding to the target posture recognition result is set as an interactive object, and action intention recognition processing is performed on target tracking data of the interactive object to obtain action intention information of the interactive object; The action intention information is mapped to game interaction control information of a vehicle game corresponding to the game interaction request, so as to control a game character in the vehicle game to perform a corresponding action through the game interaction control information.

2. The method of claim 1, wherein, The target detection result of the video frame is obtained, comprising: Obtaining brightness information of a target region in the video frame; Based on the brightness information, obtaining a brightness mean and a skewness coefficient of the target region; If the brightness mean and the skewness coefficient both satisfy a preset illumination condition, target detection processing is performed on the video frame to obtain a target detection result of the video frame.

3. The method of claim 2, wherein, The target detection processing is performed on the video frame to obtain the target detection result of the video frame, comprising: Feature processing of the video frame is performed in a channel dimension and a spatial dimension respectively by a target detection model to obtain target feature information of the video frame; the target detection model is obtained by optimizing a feature processing module of a base model through a multi-branch attention mechanism; Based on the target feature information, target detection processing is performed on the video frame by the target detection model to obtain bounding box information for the candidate object in the video frame; Based on the bounding box information, key point recognition processing is performed on a candidate object in the video frame to obtain key point information of the candidate object; According to the bounding box information and the key point information, the target detection result is obtained.

4. The method of claim 1, wherein, The action intention recognition processing on the target tracking data of the interactive object obtains action intention information of the interactive object, including: The key point information in the target tracking data of the interactive object in continuous multiple frames is subjected to action intention recognition processing for a first action to obtain first action intention information of the interactive object; If it is detected that the first action intention information does not conform to the first action, the key point information in continuous multiple frames is subjected to action intention recognition processing for a second action to obtain second action intention information of the interactive object; If it is detected that the second action intention information does not conform to the second action, the key point information in continuous multiple frames is subjected to action intention recognition processing for a third action to obtain third action intention information of the interactive object; The first action, the second action, and the third action are different.

5. The method of claim 4, wherein, The action intention recognition processing on the key point information in continuous multiple frames for a first action to obtain first action intention information of the interactive object includes: Obtaining hip position information corresponding to a hip key point and knee position information corresponding to a knee key point in the key point information; Obtaining height difference information between the hip key point and the knee key point according to the position difference between the hip position information and the knee position information; Based on the height difference information in continuous multiple frames, it is determined whether the interactive object triggers the first action to obtain the first action intention information of the interactive object.

6. The method of claim 5, wherein, The action intention recognition processing on the key point information in continuous multiple frames for the third action to obtain third action intention information of the interactive object includes: Obtaining ankle position information corresponding to an ankle key point in the key point information; Obtaining knee displacement information of the knee key point according to the knee position information in continuous multiple frames; Obtaining ankle displacement information of the ankle key point according to the ankle position information in continuous multiple frames; Based on the knee displacement information and the ankle displacement information, it is determined whether the interactive object triggers the third action to obtain the third action intention information of the interactive object.

7. The method of claim 4, wherein, The action intention recognition processing on the key point information in continuous multiple frames for the second action to obtain second action intention information of the interactive object includes: Obtaining left shoulder position information corresponding to a left shoulder key point and right shoulder position information corresponding to a right shoulder key point in the key point information; Obtaining a shoulder connecting line of the interactive object according to the left shoulder position information and the right shoulder position information; Determining an included angle between the shoulder connecting line and a target frame line in the bounding box information of the target tracking data of the interactive object; Based on the included angle in continuous multiple frames, it is determined whether the interactive object triggers the second action to obtain the second action intention of the interactive object.

8. The method of claim 1, wherein, The posture recognition result of each candidate object is obtained, including: From the target tracking data of each candidate object, a bounding box region for the candidate object is extracted; Through a posture classifier for the preset posture, posture classification processing is performed on the bounding box region of each candidate object respectively, to obtain a posture recognition result of each candidate object.

9. A game interaction control device, characterized by, The device comprises: The target detection module is configured to, in response to a game interaction request triggered by a vehicle-mounted platform of a vehicle, perform feature processing on a received video frame in channel dimension and spatial dimension respectively by an improved feature processing module in a target detection model, to obtain feature data of the video frame in the channel dimension and feature data of the video frame in the spatial dimension, and perform weighted processing on the feature data in the channel dimension and the feature data in the spatial dimension to obtain target feature information of the video frame; wherein the improved feature processing module is obtained by adding a lightweight multi-branch attention mechanism to a feature processing module of a base model; based on the target feature information, the video frame is subjected to target detection processing to obtain a target detection result of the video frame; the video frame is obtained by a shooting device of the vehicle shooting a game interaction area of the vehicle; the game interaction area is located outside the vehicle; The posture recognition module is configured to, if the target detection result indicates that the video frame contains multiple candidate objects, perform two-class processing for a preset posture on target tracking data of each candidate object by a posture classifier for the preset posture, to obtain a posture recognition result of each candidate object; the target tracking data is obtained by performing target tracking processing on the video frame; the posture recognition result is used to indicate whether the posture of the candidate object matches the preset posture; the posture classifier for the preset posture is reconstructed based on a face detection model; The action intention recognition module is configured to, if a target posture recognition result that matches the preset posture is detected in the posture recognition result, set a candidate object corresponding to the target posture recognition result as an interactive object, and perform action intention recognition processing on target tracking data of the interactive object to obtain action intention information of the interactive object; The game interaction control module is configured to map the action intention information to game interaction control information of a vehicle-mounted game corresponding to the game interaction request, to control a game character in the vehicle-mounted game to perform a corresponding action through the game interaction control information. 10.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-9. The processor executes the computer program to implement the steps of the method of any one of claims 1 to 8.

11. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 8.

Images