In-vehicle game control methods, devices, equipment and storage media

Abstract

This application relates to the field of in-vehicle entertainment system technology, and discloses an in-vehicle game control method, device, equipment, and storage medium. The method includes: acquiring an image of the vehicle's cabin space, and determining gesture images of multiple players and corresponding seat information based on the cabin space image; when the gesture images meet preset game response conditions, acquiring current game stage information, and parsing the intent of each player based on the gesture images, seat information, and current game stage information to obtain player intent; responding to game operations based on the player intent, and updating the current game stage information. This application overcomes the limitations of physical devices and the risk of dizziness through multimodal interaction technology, realizing a game control method that supports simultaneous participation by all passengers in the cabin without being limited by cabin space, greatly improving the user entertainment experience of smart cockpits.

Description

Technical Field

[0001] This application relates to the field of in-vehicle entertainment system technology, and in particular to an in-vehicle game control method, device, equipment and storage medium. Background Technology

[0002] With the development of automotive intelligence, in-vehicle entertainment systems have become an important part of enhancing the user's entertainment experience. Especially during medium and long journeys, passengers have an urgent need for entertainment methods that are highly interactive and engaging.

[0003] However, current in-vehicle entertainment systems in smart cockpits mainly interact through touch screens, voice control, or physical controllers. These interaction methods rely excessively on the screen of the gaming terminal for game interaction, and due to the special nature of the cockpit environment, they cannot support multiple passengers to participate together. Furthermore, the dynamic images on the screen can easily cause dizziness in passengers, affecting the user entertainment experience of the smart cockpit. Summary of the Invention

[0004] This application provides an in-vehicle game control method, device, equipment, and storage medium, which solves the technical problem that the interaction method of in-vehicle games in smart cockpits affects the passenger's gaming experience. By using multimodal interaction technology, it overcomes the limitations of physical devices and the risk of dizziness, and realizes a game control method that supports all passengers in the cockpit to participate simultaneously without being limited by the cabin space, greatly improving the user entertainment experience of smart cockpits.

[0005] To achieve the above objectives, the main technical solutions adopted in this application include:

[0006] In a first aspect, embodiments of this application provide an in-vehicle game control method, the method comprising:

[0007] Acquire an image of the vehicle's cabin space, and determine the gesture images of multiple players and the corresponding seat information based on the cabin space image;

[0008] When the gesture image meets the preset game response conditions, the current game stage information is obtained, and the intention of each player is analyzed according to the gesture image, the seat information and the current game stage information to obtain the player's intention;

[0009] Respond to the player's intentions with game actions and update the current game stage information.

[0010] The vehicle-mounted game control method provided in the embodiments of the present application determines gesture images and seat information by using a cabin space image, can accurately identify gestures of players in different seats, further accurately analyzes the real intentions of multiple players participating in the game in combination with current game stage information, and then performs game operation response according to the intentions of the players and updates the current game stage information in time to realize the orderly progress of the game process. In this way, compared with related technologies, players only need to interact with the game through simple gesture operations without relying on physical handles and other devices, breaking through the limitations of physical devices, and using gesture images and seat information in combination with current game stage information to analyze the intentions of each player, so that the game control is not limited by the dispersed seats in the cabin space, supports all passengers in the cabin to participate in the game at the same time, and the players do not need to look at the screen for a long time throughout the process, reducing the risk of dizziness caused by conflicts between visual signals and vestibular perception such as vehicle bumps.

[0011] Optionally, in some embodiments of the present application, the determination of the gesture images of the multiple players and the seat information corresponding to the gesture images according to the cabin space image comprises:

[0012] According to the seat distribution in the cabin, the cabin space image is divided into multiple seat areas;

[0013] The cabin space image is input into a pre-trained YOLO model to determine the gesture images according to the bounding boxes output by the YOLO model;

[0014] The target seat area corresponding to the bounding box is determined in the multiple seat areas to determine the seat information of the player according to the target seat area.

[0015] The embodiments of the present application use a YOLO (You Only Look Once) model to detect targets in the cabin space image and quickly and accurately identify gesture images of each player for subsequent intention analysis of the players. In addition, the cabin space image is divided into multiple seat areas according to the seat distribution in the cabin, and the target seat area corresponding to the identified gesture image is determined in the multiple seat areas to accurately associate the gesture image with the corresponding seat information, so as to accurately and effectively identify the gestures made by the players in different seats, support all passengers in the cabin to participate in the game at the same time without relying on physical handles and other devices, not only reduce the investment cost of physical handles and other devices, but also greatly improve the user entertainment experience of the intelligent cabin.

[0016] Optionally, in some embodiments of the present application, the determination of the target seat area corresponding to the bounding box in the multiple seat areas comprises:

[0017] determining the center coordinates of the bounding box;

[0018] obtain coordinate ranges of the plurality of seat areas, and take a seat area whose center coordinate is within the coordinate range as the target seat area.

[0019] Optionally, in some embodiments of the present application, if the gesture image is equal in number to the players, it is determined that the gesture image meets the game response condition.

[0020] The embodiments of the present application determine whether all players participating in the game have performed corresponding gesture operations by comparing the gesture image with the number of players, thereby automatically triggering the next process of the game without the players relying on physical handles or other devices for manual confirmation, improving the continuity and accuracy of game process control and greatly improving the user entertainment experience of the intelligent cockpit.

[0021] Optionally, in some embodiments of the present application, the intention analysis of each of the players according to the gesture image, the seat information and the current game stage information to obtain the player intention comprises:

[0022] generating prompt word information according to the seat information and the current game stage information;

[0023] inputting the gesture image and the prompt word information into a pre-trained multi-modal large model to extract gesture features of the gesture image according to the multi-modal large model and determine the player intention corresponding to the gesture features according to the prompt word information.

[0024] The embodiments of the present application generate prompt word information according to seat information and current game stage information to provide a parsing direction guide for actual game process for player intention analysis. Then the gesture image and the prompt word information are input into a pre-trained multi-modal large model, and the multi-modal large model is used to analyze the intention of the gesture features under the guidance of the prompt word information, which can accurately identify the player intention matching the seat information and the current game stage information, realize more intelligent gesture interaction logic, overcome the limitation of physical devices, greatly reduce the dependence of game control on the screen, effectively reduce the risk of dizziness, and comprehensively improve the entertainment interaction experience of the intelligent cockpit.

[0025] Optionally, in some embodiments of the present application, the multi-modal large model is a VLM model, and the VLM model is obtained by training in the following manner:

[0026] obtain gesture sample images, and form a plurality of input samples by respectively combining the same gesture sample image with different prompt word samples, wherein the prompt word samples include seat sample information and game stage sample information;

[0027] determine a player intention label corresponding to each of the input samples, and input each of the input samples into the VLM model for training;

[0028] adjust model parameters of the VLM model according to a comparison result of an output result of the VLM model and the player intention label, until an error rate of the output result of the VLM model and the player intention label meets a preset condition.

[0029] The embodiments of the present application train the model by using the same gesture sample image and different prompt word samples to form multiple input samples, so that the multi-modal large model can automatically make different gesture intention analysis for the same gesture feature in different game stages, and the multi-modal large model can understand the real intention of the player in different game stages, thereby significantly improving the intelligent degree of gesture intention analysis.

[0030] Optionally, in some embodiments of the present application, during the process of responding to the game operation according to the player intention, the method further comprises:

[0031] obtaining real-time driving data of the vehicle, and in a case where the real-time driving data is greater than or equal to a preset safety threshold, controlling the game operation response to be interrupted until the real-time driving data returns to be less than the preset safety threshold.

[0032] The embodiments of the present application effectively avoid potential risks that may be caused by unstable vehicle driving state and continuous game, ensure that the player plays the game under safe driving conditions, and enhance the safety of game experience in the intelligent cabin environment.

[0033] In a second aspect, the embodiments of the present application provide a vehicle-mounted game control device, the device comprising:

[0034] a gesture recognition module, configured to obtain a cabin space image of a vehicle, and determine a plurality of player gesture images and seat information corresponding to the gesture images according to the cabin space image;

[0035] an intention analysis module, configured to, in a case where the gesture images meet a preset game response condition, obtain current game stage information, and perform intention analysis on each of the players according to the gesture images, the seat information, and the current game stage information, to obtain a player intention;

[0036] a game response module, configured to respond to a game operation according to the player intention, and update the current game stage information.

[0037] The vehicle-mounted game control device provided by the embodiment of the present application determines the gesture image and seat information by using the cabin space image, can accurately identify the gestures of players in different seats, further accurately analyzes the real intentions of multiple players participating in the game in combination with the current game stage information, and then performs game operation response according to the intentions of the players and updates the current game stage information in time, so as to realize the orderly progress of the game process. Compared with the related art, the player only needs to interact with the game through simple gesture operation without relying on physical handles and other devices, thereby breaking through the limitation of physical devices. In addition, the intentions of each player are analyzed by using the gesture image and seat information in combination with the current game stage information, so that the game control is not limited by the scattered seats in the cabin space, and all passengers in the cabin can participate in the game at the same time, and the player does not need to gaze at the screen for a long time throughout the process, thereby reducing the risk of dizziness caused by the conflict between visual signals and vestibular perception such as vehicle bumping.

[0038] In a third aspect, an embodiment of the present application provides a computer device, including a memory and a processor, which are communicatively connected with each other, and the memory stores computer instructions, and the processor executes the computer instructions to perform the vehicle-mounted game control method described in the above embodiments.

[0039] In a fourth aspect, an embodiment of the present application provides a computer storage medium, which stores computer instructions, and the computer instructions are used to make a computer execute the vehicle-mounted game control method described in the above embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the specific embodiments or prior art of the present application, the drawings needed in the specific embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0041] Figure 1 One of the flowcharts of the vehicle-mounted game control method provided by an embodiment of the present application;

[0042] Figure 2 The schematic diagram of the player gesture in an embodiment of the present application;

[0043] Figure 3 The second flowchart of the vehicle-mounted game control method provided by an embodiment of the present application;

[0044] Figure 4 The third flowchart of the vehicle-mounted game control method provided by an embodiment of the present application;

[0045] Figure 5Fig. 4 is a flowchart of a vehicle game control method according to an embodiment of the present application;

[0046] Figure 6 Fig. 5 is a schematic diagram of a hardware system architecture of a vehicle game control method according to an embodiment of the present application;

[0047] Figure 7 Fig. 6 is a schematic diagram of a vehicle game control device according to an embodiment of the present application;

[0048] Figure 8 Fig. 7 is a schematic diagram of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in detail with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0050] With the development of automobile intelligence, the vehicle entertainment system has become an important part of improving user entertainment experience, especially in long-distance travel, passengers have an urgent need for interactive and high participation entertainment.

[0051] However, the current intelligent cockpit vehicle entertainment system mainly interacts through touch screens, voice control or physical handles. In some related technologies, the vehicle entertainment system supports simple voice interaction games such as "Chinese Proverbs" and the like, which are difficult to meet the depth entertainment needs of intelligent cockpit due to the single gameplay. In another related technology, the vehicle entertainment system relies on physical handles or screen touch for game interaction, but the physical handle is limited by the number of devices and the operation posture, especially for six to seven seat vehicles, the seat dispersion in the cockpit makes it inconvenient to interact, cannot support multiple seat passengers to participate together, and excessively relies on the screen of the game terminal for game interaction, which easily causes passengers to be dizzy, affecting the user entertainment experience of the intelligent cockpit.

[0052] The vehicle game control method provided in the present specification can be applied to electronic devices such as intelligent cockpit vehicle game terminals, vehicle entertainment systems, etc. Of course, the vehicle game control method provided in the present specification can also be applied in the application program running in the above-mentioned electronic devices. For example, the vehicle game control method can be applied to the game software developed for the intelligent cockpit.

[0053] According to the embodiment of the present application, a vehicle-mounted game control method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in a different order.

[0054] In the present embodiment, a vehicle-mounted game control method is provided, which can be applied to a vehicle-mounted game terminal, a vehicle-mounted entertainment system, etc. of a smart cabin, Figure 1 is a flowchart of the vehicle-mounted game control method according to the embodiment of the present application, as shown in Figure 1 , the flow includes the following steps:

[0055] Step S1, acquiring a cabin space image of a vehicle, and determining a plurality of player gesture images and seat information corresponding to the gesture images according to the cabin space image.

[0056] Specifically, the cabin space image is a panoramic image containing all seat areas collected by a camera installed in the cabin of the vehicle, for example, the cabin space image is collected by an OMS (Occupancy Monitoring System) camera in a smart cabin, which can completely collect the hand movements of players on each seat in the cabin.

[0057] The gesture image is a local image extracted from the cabin space image using a target detection algorithm. The target detection algorithm is an algorithm for locating and identifying a specific target in an image, which can accurately find the area where the player's hand is located and determine its boundary range in the cabin space image. The local image contains the hand movements of the player, so as to achieve the purpose of identifying the player's gesture.

[0058] The seat information is a position identifier determined based on the seat layout of the fixed driver seat, co-driver seat, second-row left seat, second-row right seat, etc. in the cabin space. By dividing the cabin space image into seat areas corresponding to the actual seats, the gesture image is extracted from the cabin space image by the target detection algorithm, and the seat area to which the gesture image belongs is located, so that each gesture image can be accurately associated with the corresponding player seat, providing accurate data basis for subsequent analysis of player intentions combined with seat differences.

[0059] Step S3, in the case that the gesture image meets the preset game response condition, acquiring current game stage information, and analyzing the intentions of each player according to the gesture image, the seat information and the current game stage information, to obtain the player's intention.

[0060] Specifically, in some embodiments of the present application, the current game stage information is a global variable maintained in the system memory, and the current game stage information specifically refers to the process node in which the in-vehicle game is currently located. Taking the Werewolf game as an example, the current game stage information can include a confirmation participation game stage, a player identity confirmation stage, a voting stage, a skill execution stage, and the like.

[0061] The preset game response condition can be used to determine that all players have made effective gesture operations, and only when the preset game response condition is met, the player intention analysis task based on the gesture image can be triggered, so as to ensure that the basic requirements of multi-person synchronous interaction and the normal control of the game progress are met.

[0062] Further, Figure 2 Various gestures that can be made by the player in an example of an embodiment of the present application are shown, including an "OK" gesture, a "1" gesture, a "2" gesture, a "3" gesture, a "4" gesture, and a "5" gesture. If in the confirmation participation game stage, the player makes an "OK" gesture to indicate that he / she confirms to participate in the game. In this stage, the preset game response condition can be that the number of detected gesture images does not change within a preset time, that is, after the gesture image detection in step S1 is completed, if no new or reduced gesture image is detected during the waiting period, it can be determined that the preset game response condition is met, to trigger the intention analysis of this stage, that is, the intention of the player to confirm to participate in the game. At the same time, the number of finally detected gesture images is determined as the number of players participating in the game after the intention analysis of this stage is completed.

[0063] If in other game stages, the player makes gestures Figure 2 These gestures are shown to indicate that he / she confirms the game identity, performs a voting operation, and uses a skill operation, and the like. In this stage, the preset game response condition can be that the number of gesture images is equal to the number of players, that is, if the number of gesture images is equal to the number of players, it means that each player has completed the corresponding gesture operation, and then it is determined that the gesture image meets the game response condition, to trigger the intention analysis of this stage. In this stage, a certain waiting time, such as 10 seconds, can also be set, and if there is a player who does not make a gesture operation within 10 seconds, a voice prompt is automatically triggered to prompt the corresponding player to make a gesture operation as soon as possible.

[0064] Therefore, the embodiment of the present application uses the comparison between the number of gesture images and the number of players to determine whether all players participating in the game have made the corresponding gestures, so that the next process of the game can be automatically triggered without the player relying on a physical handle or other device to make a manual confirmation, thereby improving the continuity and accuracy of the game progress control and greatly improving the user entertainment experience of the intelligent cockpit.

[0065] The embodiments of the present application perform multi-modal analysis by combining gesture image, seat information and current game stage information, wherein the seat information is bound to the game identity of the player, and the current game stage information corresponds to the game rules. In the game process, the same gesture operation made by each player can correspond to different intentions in different game stages. For example, when the player in the second row and right seat makes a "3" gesture for analysis in the voting stage, the corresponding player intention is "the player in the second row and right seat votes for the player in the No. 3 seat", and when the player in the second row and right seat makes a "3" gesture for analysis in the skill execution stage, the corresponding player intention is "the player in the second row and right seat uses a killing skill on the player in the No. 3 seat". In this way, the embodiments of the present application can accurately analyze the real intention of the player according to different game stages, thereby avoiding operation misjudgment caused by ambiguity of a single gesture in different scenes, ensuring that the instruction of each player in the multi-player interactive game can be accurately understood and matched with the current game rules, and improving the interaction accuracy of the in-vehicle game.

[0066] In step S5, a game operation response is performed according to the player intention, and the current game stage information is updated.

[0067] Specifically, the game operation response is performed based on the player intention at the business logic layer of the game main program. For example, if the intention of the player in the second row and right seat is "uses a killing skill on the player in the No. 3 seat", the business logic layer performs the operation of "records that the player uses a killing skill on the player in the No. 3 seat", and at the same time, the final operation result can be displayed on the screen of the in-vehicle game terminal after all players use the game skill, and the current game stage information is updated to the voting stage required for the next game process, thereby preparing for the player intention analysis in the voting stage.

[0068] The vehicle-mounted game control method provided in the embodiments of the present application determines the gesture image and seat information by using the cabin space image, can accurately identify the gestures of players in different seats, then accurately analyzes the real intentions of multiple players participating in the game in combination with the current game stage information, and then performs game operation response according to the intentions of the players and updates the current game stage information in time to realize the orderly progress of the game process. Especially for a smart cabin with more than 5 seats, whether it is the co-pilot or the passenger in the third row, game control can be realized through specific gesture operation, breaking the limitation of the number of handles and the distance of seats. Obviously, it can greatly improve the participation of multi-person interaction in the cabin, and let all passengers in the cabin can participate in the game. Compared with the related art, the player only needs to interact with the game through simple gesture operation without relying on physical handles and other devices, breaking the limitation of physical devices. In addition, the intentions of each player are analyzed by using the gesture image and seat information in combination with the current game stage information, so that the game control is not limited by the scattered seats in the cabin space, and all passengers in the cabin can participate in the game at the same time, and the player does not need to watch the screen for a long time throughout the process, reducing the risk of dizziness caused by the conflict between visual signals and vestibular perception such as vehicle bumping.

[0069] Figure 3 A flowchart of step S1 in the embodiments of the present application is shown, which can include the following steps:

[0070] Step S11, according to the seat distribution in the cabin, the cabin space image is divided into multiple seat areas.

[0071] Specifically, the embodiments of the present application take a seven-seater car as an example, according to the actual space distribution of the six seats in the cabin except the driving seat, the cabin space image collected by the OMS camera is divided into six corresponding specific regions (Region of Interest, ROI), and taking the ROI as a rectangular region as an example, the coordinate range [x1, x2, y1, y2] corresponding to the six ROIs is determined, wherein x1 represents the lower left corner coordinate of the ROI, x2 represents the right lower corner coordinate of the ROI, y1 represents the upper left corner coordinate of the ROI, and y2 represents the right upper corner coordinate of the ROI.

[0072] Step S13, input the cabin space image into the pre-trained YOLO model to determine the gesture image according to the bounding box (Bounding Box, bbox) output by the YOLO model.

[0073] Specifically, the YOLO model is a deep learning-based real-time target detection model, which converts target detection into a regression problem solution. After the input image passes through the network once, the position of all objects in the image, the category to which the object belongs, and the corresponding confidence probability can be obtained, so that the network structure is simple, the detection speed is greatly accelerated, and since the network has no branch, the training can be completed only once. In the application scenario of the in-vehicle entertainment system, the storage space occupied by the YOLO model is not more than 50 MB, which realizes lightweight modeling.

[0074] In addition, in some embodiments of the present application, for scenes such as light changes of day and night, and passenger body occlusion, image enhancement algorithms such as adaptive exposure adjustment and edge detection optimization are also used to improve the recognition robustness of the gesture image.

[0075] In the embodiments of the present application, the player's hand is taken as the target to be detected, the YOLO model identifies the hand in the cockpit space image, and outputs a bounding box containing hand position information, which clearly defines the specific position range of the hand in the entire cockpit space image. Then, according to the bounding box output by the YOLO model, the local image framed by the bounding box is extracted from the cockpit space image as the gesture image.

[0076] Step S15, determining the target seat area corresponding to the bounding box in the plurality of seat areas, so as to determine the seat information of the player according to the target seat area.

[0077] Further, Figure 4 A flowchart of the above step S15 in the embodiments of the present application is shown, which can include the following steps:

[0078] Step S151, determining the center coordinates of the bounding box.

[0079] Step S153, obtaining the coordinate range of the plurality of seat areas, and taking the seat area in which the center coordinates are located as the target seat area.

[0080] Specifically, the center coordinates (cx, cy) of the bounding box are determined according to the YOLO model, and it is assumed that the coordinate range of a certain seat area is [x1, x2, y1, y2]. If x1<cx<x2 and y1<cy<y2 are satisfied, the seat area is taken as the target seat area corresponding to the gesture image, and then the seat information corresponding to the target seat area is associated with the recognized gesture image.

[0081] It should be noted that in some embodiments of the present application, the game main program randomly assigns each player a game identity in the game, and binds the assigned game identity with the seat information, so that in each stage of the game, the corresponding game identity can be determined by using the seat information as one of the prompt bases for analyzing the intention of the player.

[0082] Compared with the embodiment shown in Figure 1 The embodiment of the present application uses a YOLO model to detect the target in the cockpit space image, and quickly and accurately identifies the gesture image of each player for subsequent player intention analysis. In addition, the cockpit space image is divided into multiple seat areas according to the seat distribution in the cockpit, and the target seat area corresponding to the identified gesture image is determined in the multiple seat areas, so as to accurately associate the gesture image with the corresponding seat information, to accurately and effectively identify the gesture made by the player on different seats, without relying on physical handles and other devices to support all passengers in the cockpit to participate in the game at the same time, not only reducing the investment cost of physical handles and other devices, but also greatly improving the user entertainment experience of the intelligent cockpit.

[0083] In some embodiments of the present application, the multi-modal large model is a VLM model, which is a multi-modal artificial intelligence model capable of processing and understanding image and text data at the same time. Compared with traditional single-modal models, VLM can handle more complex tasks by learning the correlation between vision and language.

[0084] Figure 5 A flowchart of step S3 in the embodiment of the present application is shown, which can include the following steps:

[0085] Step S31, generating prompt word information according to seat information and current game stage information.

[0086] Specifically, first, the input data structure of the VLM model is constructed, which includes the following strings:

[0087] “original_image: Optional[bytes] = None # represents the original image byte data of the gesture image

[0088] gesture_bbox: tuple = () # represents the bounding box corresponding to the gesture image

[0089] game_phase: str ="" # represents the current game stage information

[0090] position: str ="" # represents the seat information corresponding to the gesture image

[0091] To guide the VLM model to accurately analyze the intention, a prompt word construction command "prompt = build_vlm_prompt(vlm_input)" is called to obtain prompt word information, which includes a seat information corresponding string "position: str ="" and a current game phase information corresponding string "game_phase: str ="". The seat information is used to prompt the player's game identity, and the current game phase information is used to prompt the player's real intention in the current game phase.

[0092] In step S33, the gesture image and the prompt word information are input into the pre-trained multi-modal large model to extract gesture features of the gesture image according to the multi-modal large model, and to determine the player intention corresponding to the gesture features according to the prompt word information.

[0093] Specifically, the embodiment of the present application calls open source VLM models such as LLaA and MiniGPT-4 to realize intention analysis.

[0094] Firstly, the VLM model is initialized and set, wherein the model name is defined and the corresponding preprocessor is loaded, which is used for subsequent format processing of the input data to meet the input specification of the model. Then, the VLM model is loaded and configured, such as setting the data type to half-precision floating point and optimizing the CPU memory.

[0095] Subsequently, the pre-trained VLM model converts the original image byte data of the input gesture image into a Python Imaging Library (PIL) image, so that the gesture image meets the requirements of the VLM model for image input format, facilitating the VLM model to extract gesture features, thereby obtaining gesture features. The gesture features and the prompt word information are fused and inferred to map the gesture features to specific player intentions, ensuring the accurate association of the gesture features with the seat information and the game phase information contained in the prompt word, so that the output player intention matches the current game phase and the player identity.

[0096] Compared with the embodiment shown in Figure 1 Compared with the embodiment shown in The embodiment of the present application generates prompt word information according to the seat information and the current game phase information, providing a parsing direction guide for the actual game process for player intention analysis. Then, the gesture image and the prompt word information are input into the pre-trained multi-modal large model, and the multi-modal large model is used to analyze the intention of the gesture features under the guidance of the prompt word information, which can accurately identify the player intention matching the seat information and the current game phase information, realize more intelligent gesture interaction logic, overcome the limitation of physical devices, greatly reduce the dependence of game control on the screen, effectively reduce the risk of dizziness, and comprehensively improve the entertainment interaction experience of the intelligent cockpit.

[0097] Further, in some embodiments of the present application, the VLM model is trained by the following way:

[0098] First, gesture sample images are obtained, and the same gesture sample image is combined with different prompt word samples to form multiple input samples, wherein the prompt word sample includes seat sample information and game stage sample information.

[0099] Specifically, gesture sample images of "OK" gesture, "1" gesture, "2" gesture, "3" gesture, "4" gesture and "5" gesture are collected respectively, and each gesture sample image is combined with different prompt word samples. The seat sample information corresponds to the game identity, such as civilian, werewolf, witch, etc., and the game stage sample information includes confirming the participation in the game stage, the player identity confirmation stage, the voting stage, the skill execution stage, etc. The above gesture sample images are arranged and combined with different seat sample information and game stage sample information to obtain multiple input samples. For example, for the gesture sample image of "1" gesture, the gesture sample image is combined with "1" gesture, "wolf" as seat sample information, and "voting stage" as game stage sample information to obtain an input sample, the gesture sample image is combined with "1" gesture, "wolf" as seat sample information, and "skill execution stage" as game stage sample information to obtain another input sample, the gesture sample image is combined with "1" gesture, "civilian" as seat sample information, and "voting stage" as game stage sample information to obtain another input sample, and so on.

[0100] Subsequently, the player intention label corresponding to each input sample is determined, and each input sample is input into the VLM model for training. The model parameters of the VLM model are adjusted according to the comparison result of the output result of the VLM model and the player intention label until the error rate of the output result of the VLM model and the player intention label meets the preset condition.

[0101] Specifically, the correct player intention label is labeled for each input sample, for example, for the input sample of gesture sample image "1" gesture, seat sample information "wolf", and game stage sample information "voting stage", the player intention label labeled for it is "voting for the player in No. 1 seat", and for the input sample of gesture sample image "1" gesture, seat sample information "wolf", and game stage sample information "skill execution stage", the player intention label labeled for it is "using the killing skill on the player in No. 1 seat".

[0102] Therefore, the embodiment of the application trains the model by using the same gesture sample image and different prompt sample groups to form multiple input samples, so that the multi-modal large model can automatically analyze different gesture intentions for the same gesture feature in different game stages, so that the multi-modal large model can understand the real intention of the player in different game stages, and the intelligent degree of gesture intention analysis is significantly improved.

[0103] In some embodiments of the application, in the process of responding to the game operation according to the player's intention, the method further comprises: acquiring real-time driving data of the vehicle, and in the case that the real-time driving data is greater than or equal to a preset safety threshold, controlling the game operation response to be interrupted until the real-time driving data is restored to be less than the preset safety threshold.

[0104] Specifically, the real-time driving data includes real-time driving speed, steering angular speed, brake pedal depth, throttle opening and the like of the vehicle. If the real-time driving data is greater than or equal to the preset safety threshold, it indicates that the current driving road condition of the vehicle is unstable, and the game process is interrupted at this time. For example, when the real-time driving speed of the vehicle is greater than 60km / h and the steering angular speed is continuously greater than 30°, the acceleration is greater than 0.5g, the game is automatically paused, and the current game progress is temporarily retained at this time, and the game is restored after the road condition is stable, avoiding the game experience from being reduced due to restart.

[0105] The embodiment of the application effectively avoids the potential risks that may be caused by continuous game in unstable vehicle driving state, ensures that the player plays the game under safe driving conditions, and enhances the safety of game experience in the intelligent cabin environment.

[0106] Further, Figure 6 The hardware system architecture of the above vehicle-mounted game control method is shown in FIG. 1. Figure 6 As shown in FIG. 1, in some embodiments of the application, the above vehicle-mounted game control method is implemented by a SOC chip, which integrates a game main program and has the ability to support the integration of YOLO model and VLM model, and at the same time needs to be provided with a display screen, at least one OMS camera and connected with the SOC chip, and the intelligent cabin needs to have networking function. The OMS camera is used to collect the cabin space image and transmit it to the SOC chip, the player's intention is analyzed based on the cabin space image and the current game stage information by the YOLO model and the VLM model integrated in the SOC chip, and the analyzed player's intention is transmitted to the game main program for game operation response, at the same time, the game main program updates the current game stage information, and the game process interaction is realized through the display screen, such as displaying the voting decision result, prompting the next game stage and reminding the player to perform gesture operation and the like.

[0107] In some embodiments of the present application, the display time of the display screen is limited to no more than 3 seconds to reduce visual dependence, match the vestibular perception in vehicle driving, effectively reduce the risk of dizziness, and comprehensively improve the entertainment interaction experience of the intelligent cockpit.

[0108] Correspondingly, please refer to Figure 7 The embodiments of the present application provide a vehicle-mounted game control device, which comprises:

[0109] The gesture recognition module 100 is configured to acquire a cabin space image of the vehicle, and determine a plurality of player gesture images and seat information corresponding to the gesture images according to the cabin space image. For details, please refer to step S1.

[0110] The intention analysis module 200 is configured to acquire current game stage information when the gesture images meet a preset game response condition, and perform intention analysis on each player according to the gesture images, the seat information and the current game stage information, to obtain player intentions. For details, please refer to step S3.

[0111] The game response module 300 is configured to perform game operation response according to the player intentions, and update the current game stage information. For details, please refer to step S5.

[0112] In some embodiments of the present application, the gesture recognition module 100 comprises:

[0113] The image division unit 110 is configured to divide the cabin space image into a plurality of seat regions according to the seat distribution in the cabin.

[0114] The target detection unit 120 is configured to input the cabin space image into a pre-trained YOLO model, to determine the gesture images according to the bounding boxes output by the YOLO model.

[0115] The seat determination unit 130 is configured to determine a target seat region corresponding to the bounding box in the plurality of seat regions, to determine the seat information of the player according to the target seat region.

[0116] In some embodiments of the present application, the seat determination unit 130 can further comprise:

[0117] The coordinate determination subunit 131 is configured to determine the center coordinates of the bounding box.

[0118] The region determination subunit 132 is configured to acquire the coordinate range of the plurality of seat regions, and take the seat region in which the center coordinates are located as the target seat region.

[0119] In some embodiments of the present application, the intention analysis module 200 comprises:

[0120] The prompt word unit 210 is configured to generate prompt word information according to the seat information and the current game stage information.

[0121] The multi-modal analysis unit 220 is configured to input the gesture image and the prompt word information into a pre-trained multi-modal large model, to extract gesture features of the gesture image according to the multi-modal large model, and to determine a player intention corresponding to the gesture features according to the prompt word information.

[0122] In some embodiments of the present application, the device further comprises a model training module 400, specifically configured to:

[0123] Obtain gesture sample images, and form a plurality of input samples by respectively combining the same gesture sample images with different prompt word samples, wherein the prompt word samples comprise seat sample information and game stage sample information.

[0124] Determine player intention labels corresponding to the input samples, and input the input samples into the VLM model for training.

[0125] Adjust model parameters of the VLM model according to a comparison result of an output result of the VLM model and the player intention labels, until an error rate of the output result of the VLM model and the player intention labels satisfies a preset condition.

[0126] In some embodiments of the present application, the device further comprises a response interruption module 500, specifically configured to obtain real-time driving data of the vehicle, and control interruption of a game operation response in a case where the real-time driving data is greater than or equal to a preset safety threshold, until the real-time driving data recovers to be less than the preset safety threshold.

[0127] Further function descriptions of the above-mentioned modules and units are the same as those of the corresponding embodiments, and will not be repeated here.

[0128] The vehicle-mounted game control device in the present embodiment is presented in the form of a functional unit. The unit herein refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and a memory executing one or more software or fixed programs, and / or other devices that can provide the above-mentioned functions.

[0129] Please refer to Figure 8 , Figure 8 is a structural schematic diagram of a computer device provided by the embodiments of the present application, such as Figure 8As shown, the computer device includes one or more processors 10, memory 20, and interfaces 30 for the various components to communicate with one another. The various components communicate through the use of the various buses, and can be mounted on a common motherboard or in other manners as appropriate. The processor 10 can process instructions for execution within the computer device, including instructions stored in the memory 20 or elsewhere within the computer device. In some optional embodiments, multiple processors 10 and / or multiple buses can be employed as appropriate, along with multiple memories and types of memory. Also, multiple computer devices can be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or a multi-processor system). Figure 8 The processor 10 is taken as an example in the embodiments.

[0130] The processor 10 can be a central processing unit, a network processor, or a combination thereof. The processor 10 can further include a hardware chip. The hardware chip can be an application specific integrated circuit, a programmable logic device, or a combination thereof. The programmable logic device can be a complex programmable logic device, a field programmable logic device, a general array logic, or any combination thereof.

[0131] The memory 20 stores instructions that are executable by the at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiments.

[0132] The memory 20 can include a program region and a data region. The program region can store an operating system and an application program required by at least one function. The data region can store data created according to the use of the computer device, and the like. In addition, the memory 20 can include a high-speed random access memory, and can further include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some optional embodiments, the memory 20 can optionally include a memory disposed remotely with respect to the processor 10, and these remote memories can be connected to the computer device through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0133] The memory 20 can include a volatile memory, such as a random access memory, and can also include a non-volatile memory, such as a flash memory, a hard disk, or a solid state disk. The memory 20 can further include a combination of the above-mentioned memories.

[0134] The computer device further includes a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0135] The method according to the embodiments of the present application can be implemented in hardware, firmware, or recorded in a storage medium, or be implemented as computer code stored in a storage medium or downloaded from a remote storage medium and stored in a local storage medium to be executed by a local computer, so that the method described herein can be processed by such software on a storage medium using a general-purpose computer, a special-purpose processor, or programmable or special-purpose hardware. The storage medium can be a magnetic disk, an optical disk, a read-only memory, a random access memory, a flash memory, a hard disk, or a solid-state disk, etc. Further, the storage medium can also include a combination of the above-mentioned types of memories. It can be understood that the computer, processor, microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the method shown in the above embodiments.

[0136] The embodiments of the present application provide a computer program product, which includes computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to enable the computer device to perform the method of any of the embodiments of the present application.

[0137] Although the embodiments of the present application are described in conjunction with the accompanying drawings, various modifications and changes can be made by those skilled in the art without departing from the spirit and scope of the present application, and such modifications and changes fall within the scope defined by the appended claims.

[0138] It can be understood that, before using the technical solutions disclosed in the embodiments of the present disclosure, the type of personal information involved in the present disclosure, the use range, the use scenario, etc. should be informed to the user and the authorization of the user should be obtained through appropriate means according to relevant laws and regulations.

[0139] For example, in response to receiving the active request of the user, prompt information is sent to the user to explicitly prompt the user that the operation requested to be performed will require obtaining and using the personal information of the user. Thus, the user can voluntarily choose whether to provide the personal information to the software or hardware such as the electronic device, application program, server, or storage medium that performs the operation of the technical solutions of the present disclosure according to the prompt information.

[0140] As an optional but not limited implementation manner, in response to receiving the active request of the user, the manner of sending the prompt information to the user can be, for example, the manner of a pop-up window, and the prompt information can be presented in the form of text in the pop-up window. In addition, the pop-up window can also carry a selection control for the user to select "agree" or "disagree" to provide the personal information to the electronic device.

[0141] It can be understood that the above notification and obtaining user authorization process is only illustrative, and does not limit the implementation of the present disclosure, and other ways that meet the relevant laws and regulations can also be applied to the implementation of the present disclosure.

[0142] It can be understood that the data involved in the technical solution (including but not limited to the data itself, the acquisition or use of the data) should comply with the requirements of the relevant laws and regulations and the relevant provisions.

[0143] It can be understood that in the specific embodiments of the present application, data related to user information, seat information, navigation data, etc. are involved. When the above embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data need to comply with relevant national and regional laws, regulations and standards.

[0144] The system, device, module or unit illustrated in the above embodiments can be specifically implemented by a computer chip or entity, or by a product with certain functions. A typical implementation device is a computer. Specifically, the computer may, for example, be a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0145] For the convenience of description, the above device is described as various units divided by functions. Of course, the functions of each unit can be implemented in the same or more software and / or hardware in the implementation of the present application.

[0146] Those skilled in the art will understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0147] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 The flowchart and / or block diagram in the variation of the present application can be implemented by computer program instructions. These program instructions can be provided to a processor of a general-purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions which are executed on the computer or other programmable apparatus generate a device implemented process for implementing the functions specified in the flowchart and / or block diagram block or blocks. Figure 1 These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device that implements the function specified in the flowchart and / or block diagram block or blocks.

[0148] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart and / or block diagram block or blocks. Figure 1 The flowchart and / or block diagram in the variation of the present application can be implemented by computer program instructions. These program instructions can be provided to a processor of a general-purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions which are executed on the computer or other programmable apparatus generate a device implemented process for implementing the functions specified in the flowchart and / or block diagram block or blocks. Figure 1 These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device that implements the function specified in the flowchart and / or block diagram block or blocks.

[0149] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart and / or block diagram block or blocks. Figure 1 The flowchart and / or block diagram in the variation of the present application can be implemented by computer program instructions. These program instructions can be provided to a processor of a general-purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions which are executed on the computer or other programmable apparatus generate a device implemented process for implementing the functions specified in the flowchart and / or block diagram block or blocks. Figure 1 These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device that implements the function specified in the flowchart and / or block diagram block or blocks.

[0150] It should be noted that the terms "comprising", "including", or any other variations thereof are intended to cover a non-exclusive inclusion, such that processes, methods, articles, or apparatuses that comprise a list of elements do not include only those elements in the list, but can also include other elements not expressly listed or inherent to such processes, methods, articles, or apparatuses. Without further limitation, an element preceded by "comprises a" does not, without more constraints, foreclose additional identical elements of the comprising process, method, article, or apparatus.

[0151] Each of the embodiments in the present specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the difference from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.

[0152] The above merely provides an example of the present application, but is not intended to limit the present application. The present application can have various modifications and changes for those skilled in the art. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the scope of claims of the present application.

[0153] Although the embodiments of the present application are described with reference to the drawings, various modifications and changes can be made by those skilled in the art without departing from the spirit and scope of the present application, and such modifications and changes shall fall within the scope defined by the appended claims.

Claims

1. A vehicle-mounted game control method, characterized by, The method comprises: acquiring a cabin space image of a vehicle, and determining gesture images of multiple players and seat information corresponding to the gesture images according to the cabin space image; in a case where the gesture images satisfy a preset game response condition, acquiring current game stage information, and performing intention analysis on each of the players according to the gesture images, the seat information and the current game stage information to obtain player intentions; performing game operation response according to the player intentions, and updating the current game stage information; the intention analysis on each of the players according to the gesture images, the seat information and the current game stage information to obtain player intentions comprises: generating prompt word information according to the seat information and the current game stage information; inputting the gesture images and the prompt word information into a pre-trained multi-modal large model to extract gesture features of the gesture images according to the multi-modal large model, and determining the player intentions corresponding to the gesture features according to the prompt word information.

2. The method of claim 1, wherein, the determination of the gesture images of multiple players and the seat information corresponding to the gesture images according to the cabin space image comprises: dividing the cabin space image into multiple seat regions according to seat distribution in the cabin; inputting the cabin space image into a pre-trained YOLO model to determine the gesture images according to a bounding box output by the YOLO model; determining a target seat region corresponding to the bounding box in the multiple seat regions to determine the seat information of the players according to the target seat region.

3. The method of claim 2, wherein, the determination of the target seat region corresponding to the bounding box in the multiple seat regions comprises: determining a center coordinate of the bounding box; acquiring coordinate ranges of the multiple seat regions, and regarding a seat region in which the center coordinate is located as the target seat region.

4. The method of claim 1, wherein, if the number of the gesture images is equal to the number of the players, it is determined that the gesture images satisfy the game response condition.

5. The method of claim 1, wherein, the multi-modal large model is a VLM model, and the VLM model is trained in the following manner: acquiring gesture sample images, and forming multiple input samples by combining the same gesture sample image with different prompt word samples, wherein the prompt word samples comprise seat sample information and game stage sample information; determining player intention labels corresponding to each of the input samples, and inputting each of the input samples into the VLM model for training; adjusting model parameters of the VLM model according to a comparison result of an output result of the VLM model and the player intention labels until an error rate of the output result of the VLM model and the player intention labels satisfies a preset condition.

6. The method of claim 1, wherein, in the process of performing game operation response according to the player intentions, the method further comprises: acquiring real-time driving data of the vehicle, and controlling the game operation response to be interrupted in a case where the real-time driving data is greater than or equal to a preset safety threshold until the real-time driving data returns to be less than the preset safety threshold.

7. An in-vehicle game control device characterized by comprising: the device comprises: The gesture recognition module is configured to acquire a cabin space image of the vehicle, and determine a plurality of player gesture images and seat information corresponding to the gesture images according to the cabin space image; The intention analysis module is configured to, when the gesture images meet a preset game response condition, acquire current game stage information, and perform intention analysis on each of the players according to the gesture images, the seat information, and the current game stage information, to obtain player intentions; The game response module is configured to perform game operation response according to the player intentions, and update the current game stage information; The intention analysis module includes: A prompt word unit configured to generate prompt word information according to the seat information and the current game stage information; A multi-modal analysis unit configured to input the gesture images and the prompt word information into a pre-trained multi-modal large model, to extract gesture features of the gesture images according to the multi-modal large model, and determine the player intentions corresponding to the gesture features according to the prompt word information.

8. A computer device, comprising: The vehicle-mounted game control method includes: A memory and a processor, which are communicatively connected, the memory stores computer instructions, and the processor executes the computer instructions to perform the vehicle-mounted game control method according to any one of claims 1 to 6.

9. A computer storage medium, characterized in that The computer readable storage medium stores computer instructions, and the computer instructions are used to make the computer execute the vehicle-mounted game control method according to any one of claims 1 to 6.

Images