Method and system for controlling a vehicle component and computer program product

Abstract

This application proposes a method for controlling vehicle components, comprising: recognizing a user's voice input via a front-seat infotainment system running on a first controller to obtain a voice recognition result; recognizing a user's gesture input via a rear-seat infotainment system running on a second controller to obtain a gesture recognition result, wherein the rear-seat infotainment system sends the gesture recognition result to the front-seat infotainment system; and matching and fusing the voice recognition result and the gesture recognition result to generate a control command. This application also relates to a corresponding system and computer program product for controlling vehicle components. Thus, a cockpit control architecture with dual controllers and dual operating systems can be used to achieve multimodal command control of vehicle components combining gestures and voice.

Description

Technical Field

[0001] This application relates to the field of vehicles, and in particular to a method for controlling vehicle components, a system for controlling vehicle components, and a computer program product. Background Technology

[0002] With the rapid development of vehicle intelligence technology, the human-machine interaction of vehicles is constantly being upgraded, providing drivers and passengers with a more convenient and comfortable driving experience.

[0003] Currently, it is known that various vehicle components can be controlled via voice commands. Drivers or passengers can control multiple electronic components and functional modules in the vehicle by issuing voice commands. This control method eliminates the need for manual operation, providing a degree of convenience for users. However, the aforementioned voice-command-based vehicle component control method still has many shortcomings and cannot fully meet the control needs of different drivers and passengers. For example, drivers and passengers may have limited knowledge of vehicle components, which may make it difficult for them to issue accurate voice commands. As vehicle configurations become increasingly sophisticated, the types of electronic components in vehicles are also increasing, with some components having specialized functions. The naming conventions for components also differ across vehicle models, and ordinary drivers and passengers often cannot fully grasp the accurate names of all vehicle components and their corresponding voice control commands, thus failing to achieve precise control of target components via voice commands. Furthermore, if the driver or passenger's pronunciation is not standard, their speech is too fast, or they have an accent, the vehicle's voice recognition system is prone to recognition errors, failing to accurately respond to commands and leading to control failure.

[0004] To address the inconvenience of rear-seat passengers operating vehicle components, some existing vehicles are equipped with rear-seat screens. These screens allow rear passengers to easily control vehicle components via touchscreens, partially compensating for the lack of convenient voice control for rear-seat passengers. However, the control method of rear-seat screens still has limitations, requiring manual operation by passengers, making the process relatively cumbersome.

[0005] Therefore, existing technologies still have shortcomings in terms of vehicle component control methods, and there is an urgent need for a more convenient, efficient, and easy-to-operate method for controlling vehicle components. Summary of the Invention

[0006] The purpose of this application is to provide an improved method for controlling vehicle components, as well as a corresponding system and computer program product, in order to at least partially overcome the shortcomings of the prior art.

[0007] According to a first aspect of this application, a method for controlling a vehicle component is provided, comprising the following steps: -S1, using the front-row infotainment system running on the first controller to recognize the user's voice input to obtain a voice recognition result containing the target control operation information; -S2, using a rear-seat infotainment system running on a second controller, the user's gesture input is recognized to obtain a gesture recognition result containing target control object information, the target control object information being used to indicate the vehicle component to be controlled, wherein the rear-seat infotainment system sends the gesture recognition result to the front-seat infotainment system; and -S3 matches the speech recognition results with the gesture recognition results and merges the matched speech recognition results with the gesture recognition results to generate control instructions for controlling the target object to perform target control operations.

[0008] The method according to the exemplary embodiments of this application enables multimodal command control of vehicle components using a dual-controller, dual-operating-system cockpit control architecture, combining gestures and voice commands. Passengers in the rear seats can also operate vehicle components via both voice and gesture commands. This control method aligns with natural human behavior, helping to lower the barrier to entry for controlling vehicle components and improve ease of operation.

[0009] Vehicle components may include various parts, components and / or functional modules of a vehicle.

[0010] In one exemplary embodiment, the rear infotainment system includes a system user space and multiple sub-user spaces corresponding to different cabin areas. In step S2, the voice dialogue adapter of the sub-user space acquires the gesture recognition result associated with the corresponding cabin area, and the system user space sends the source information associated with the sub-user space and the gesture recognition result from the sub-user space to the front infotainment system.

[0011] In an exemplary embodiment, in step S3, the front-row infotainment system matches the speech recognition results and gesture recognition results based at least on the source information and fuses the matched speech recognition results and gesture recognition results to generate control instructions for controlling the target control object to perform target control operations.

[0012] In one exemplary embodiment, the plurality of sub-user spaces of the rear-seat infotainment system independently determine the location information of the target control object based on gesture input. This helps to more accurately identify the vehicle component to be controlled.

[0013] In one exemplary embodiment, different sub-user spaces determine the location information of the target controlled object based on gesture input according to different association relationships. Assigning different association relationships to different sub-user spaces helps to improve the accuracy of identifying the vehicle component to be controlled in a convenient manner.

[0014] In one exemplary embodiment, different sub-user spaces utilize different neural network models to determine the position information of the target controlled object based on gesture input. This helps to accurately and efficiently determine the position of the target controlled object. Different neural network models may differ in aspects such as training data, network structure, training strategies, and / or feature extraction methods.

[0015] In one exemplary embodiment, the vehicle is provided with a plurality of rear screens mounted on the rear side of the front seats, and the plurality of sub-user spaces of the rear infotainment system are respectively associated with one of the plurality of screens. The gesture recognition result acquired by each sub-user space is generated by image recognition of an image captured by a camera device integrated in the rear screen associated with the sub-user space.

[0016] In one exemplary embodiment, the method includes: if the front infotainment system only acquires a voice recognition result and does not receive a gesture recognition result that matches the voice recognition result, then determining a target control object based on the cabin area where the user who issued the voice input is located.

[0017] In one exemplary embodiment, the voice recognition result obtained by means of the front infotainment system in step S1 also includes additional target control object information obtained by recognizing the user's voice input. In step S3, the front infotainment system fuses the additional target control object information of the voice recognition result with the target control object information of the matching gesture recognition result to determine the vehicle component to be controlled. This helps to determine the vehicle component to be controlled more accurately and reliably.

[0018] In one exemplary embodiment, fusing the additional target control object information with the target control object information includes the following sub-steps: - Determine whether the additional target control object information conflicts with the target control object information; - If the additional target control object information conflicts with the target control object information, the target control object is determined solely based on the target control object information, wherein the target control object may optionally be a vehicle component that conforms to the target control object information and has the highest confidence level; and - If the additional target control object information does not conflict with the target control object information, then the target control object is determined based on the target control object information and the additional target control object information, wherein the target control object is a vehicle component that conforms to the target control object information and the additional target control object information.

[0019] Optionally, if the additional target control object information does not conflict with the target control object information, the type of target control object to be controlled can be determined by the additional target control object information, and the location of the target control object to be controlled can be determined by the target control object information.

[0020] In an exemplary embodiment, in step S3, if the front-row infotainment system receives valid gesture recognition results from two or more sub-user spaces, and the gesture recognition results all have matching valid voice recognition results, and the gesture recognition results and their matching voice recognition results from the two or more sub-user spaces conflict with each other, then the sub-user space with higher priority is selected from the two or more sub-user spaces, and a control command for controlling the target control object to perform target control operations is generated based on the gesture recognition results from the higher-priority sub-user space and the matching voice recognition results.

[0021] In one exemplary embodiment, the method further includes: acquiring the identity information of the user providing gesture input via facial recognition using a rear-seat infotainment system and sending it to a front-seat infotainment system. In step S3, the front-seat infotainment system determines whether to generate a control command based on the gesture recognition result associated with the identity information, according to the operation permissions and / or priority corresponding to the identity information.

[0022] In one exemplary embodiment, the method further includes: identifying the user's lip movement characteristics using a rear-seat infotainment system to obtain lip movement recognition results and sending them to a front-seat infotainment system. In step S3, the front-seat infotainment system determines the location of the user issuing the voice command based at least on the lip movement recognition results, for matching the voice recognition results with the gesture recognition results.

[0023] In one exemplary embodiment, the front infotainment system and the rear infotainment system are different types of operating systems.

[0024] The front-seat infotainment system can be a Linux operating system. Alternatively or additionally, the rear-seat infotainment system may be an Android operating system.

[0025] In one exemplary embodiment, structured data is transmitted between the front-seat infotainment system and the rear-seat infotainment system via a communication middleware layer that combines Thrift architecture and GRPC architecture.

[0026] In one exemplary embodiment, the target control object includes the vehicle's ambient lighting.

[0027] Target control operations may include, for example, adjusting color, brightness, on / off status, and / or flashing mode.

[0028] According to a second aspect of this application, a computer program product is provided, comprising computer program control instructions, wherein, when executed by one or more processors, the computer program control instructions cause the one or more processors to perform a method for controlling vehicle components according to an exemplary embodiment of this application.

[0029] According to a third aspect of this application, a system for controlling vehicle components is provided, comprising: a first controller for operating a front-seat infotainment system; and a second controller for operating a rear-seat infotainment system, wherein the first controller and the second controller are communicatively connected to each other and configured to perform a method for controlling vehicle components according to an exemplary embodiment of this application. Attached Figure Description

[0030] The principles, features, and advantages of this application will be better understood below with reference to the accompanying drawings. The drawings include: Figure 1 A system for controlling vehicle components according to an exemplary embodiment of this application is illustrated schematically; Figure 2 The illustration schematically depicts a method for controlling vehicle components according to exemplary embodiments of this application; Figure 3 A flowchart illustrating step S3 of a method for controlling vehicle components according to an exemplary embodiment of this application is shown schematically. Figure 4 An architectural block diagram of a vehicle infotainment system according to an exemplary embodiment of this application is schematically shown; Figure 5 The illustration schematically depicts a user located in the left rear seat controlling vehicle components via both voice and gesture commands in an exemplary embodiment according to this application; and Figure 6 This illustration schematically shows a user located in the right rear seat controlling vehicle components via both voice and gesture commands in an exemplary embodiment according to this application.

[0031] List of reference numerals Detailed Implementation

[0032] To make the technical problems to be solved, the technical solutions, and the beneficial technical effects of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and several exemplary embodiments. It should be understood that the specific embodiments described herein are only for explaining this application and are not intended to limit the scope of protection of this application.

[0033] Figure 1A system for controlling vehicle component 10 according to an exemplary embodiment of this application is illustrated schematically. Figure 2 The method for controlling vehicle component 10 according to an exemplary embodiment of the present application is illustrated schematically.

[0034] like Figure 1 As shown, the vehicle has front seats 11 and rear seats 12. To facilitate vehicle operation or entertainment for the driver or passengers, the vehicle may have one or more front screens 13. The front screens 13 may be located, for example, in the center console area, making them easy for the front-seat user (driver or passenger) to operate and view. The vehicle may also have multiple rear screens 14 mounted on the rear side of the front seats 11. Users in the rear seats 12 can easily and quickly operate and use the rear screens 14. The rear screens 14 may be equipped with a camera device 15 configured to capture images of the users in the rear seats 12.

[0035] The vehicle is also equipped with microphones 16, particularly multiple microphones (or microphone arrays) 16, for receiving voice commands from the user. The microphones 16 may be distributed in multiple different locations within the vehicle. For example, microphones 16 may be located on the roof, A-pillars, seat headrests, and / or armrests. Thus, regardless of the user's location within the vehicle cabin, their voice commands can be clearly captured.

[0036] The system for controlling vehicle components 10 may include a first controller 21 and a second controller 22, wherein the first controller 21 is used to operate a front-seat infotainment system and the second controller 22 is used to operate a rear-seat infotainment system. The front-seat infotainment system may be configured as the control center of the front-seat infotainment system. The rear-seat infotainment system may be configured as the control center of a rear-seat entertainment system (RSE). The rear-seat infotainment system is a rear-seat in-vehicle infotainment configuration designed to enhance the riding experience of rear passengers. The rear-seat infotainment system may include or be connected to multiple rear-seat screens 14. The first controller 21 and the second controller 22 may be communicatively connected to each other and configured to perform methods for controlling vehicle components 10. According to embodiments of the present invention, the first controller and the second controller may run different operating systems (e.g., Linux and Android operating systems, respectively). Alternatively, the front-seat infotainment system and the rear-seat infotainment system may run on different operating systems.

[0037] See Figure 2 The method for controlling vehicle component 10 includes steps S1, S2 and S3.

[0038] In step S1, the user's voice input is recognized by means of the front-seat infotainment system running on the first controller 21 to obtain a voice recognition result containing target control operation information. The target control operation information is used to instruct the user on the operation that the vehicle component 10 wants to perform.

[0039] In step S2, the rear-seat infotainment system, running on the second controller 22, recognizes the user's gesture input to obtain a gesture recognition result containing target control object information. The target control object information is used to indicate the vehicle component 10 to be controlled. The rear-seat infotainment system sends the gesture recognition result to the front-seat infotainment system.

[0040] In step S3, the speech recognition results and gesture recognition results are matched and the matched speech recognition results and gesture recognition results are fused to generate control instructions for controlling the target control object to perform target control operations.

[0041] The method according to the exemplary embodiments of this application enables multimodal command control of vehicle component 10 using a dual-controller, dual-operating-system front and rear-seat infotainment and cockpit control architecture, combining gestures and voice commands. Rear-seat passengers can also conveniently and reliably operate vehicle component 10 via both voice and gesture commands. This control method aligns with natural human behavior, helping to lower the barrier to entry for controlling vehicle component 10 and improve operational convenience. Even if the user does not know the name of the vehicle component 10 to be controlled, they can still operate it. For users in different positions, the vehicle component 10 they wish to control can be accurately identified.

[0042] The method described above for controlling vehicle component 10 can fully utilize the characteristics and advantages of each operating system in the dual controller dual operating system, and can achieve low-latency, high-synchronization system communication and cooperation between different operating systems.

[0043] The target control object includes, for example, vehicle equipment or components such as reading lights and ambient lights. According to an exemplary embodiment of this application, the target control operation may include adjusting color, brightness, on / off state, and / or flashing mode.

[0044] As an example, when a passenger sitting on the left rear seat of the vehicle points upwards to the left rear reading light 17 and says "turn this on," a control command to turn on the left rear reading light 17 can be generated by the method of controlling the vehicle component 10 according to this application.

[0045] Specifically, the front-row infotainment system running on the first controller 21 can obtain a voice recognition result by recognizing the user's voice input "turn this on". This voice recognition result may include target control operation information, indicating that the user's target control operation is "turn on". For example, through sound source identification, the front-row infotainment system can also determine the location of the user who issued the voice input.

[0046] The rear infotainment system running on the second controller 22 can recognize the user's gesture input to obtain gesture recognition results containing information about the target controlled object.

[0047] The rear infotainment system may include a system user space 221 and multiple sub-user spaces corresponding to different cabin areas. In step S2, the voice dialogue adapter (SDSAdapter) of the sub-user space acquires the gesture recognition result associated with the corresponding cabin area. The system user space 221 sends the source information associated with the sub-user space and the gesture recognition result from the sub-user space to the front infotainment system.

[0048] In step S3, the front infotainment system can match the voice recognition results and gesture recognition results based at least on the source information, and fuse the matched voice recognition results and gesture recognition results to generate control commands for controlling the target controlled object to perform target control operations. As an example, the rear infotainment system may have a system user space 221 and two sub-user spaces. The two sub-user spaces may include a first sub-user space 222 (sub-user0) and a second sub-user space 223 (sub-user1) corresponding to the left rear cabin area and the right rear cabin area, respectively. The SDSAdapter of sub-user0 can obtain the gesture recognition results associated with the left rear cabin area. The system user space 221 sends the source information associated with the sub-user space and the gesture recognition results from the sub-user space to the front infotainment system.

[0049] Source information may include, for example, sub-user space identifiers (e.g., sub-user0, sub-user1), which indicate which sub-user space the gesture recognition result comes from.

[0050] Optionally, the source information may be represented as information used to indicate the cabin area corresponding to the sub-user space, such as the left rear cabin area and the right rear cabin area.

[0051] The front infotainment system can receive gesture recognition results and corresponding source information from the first sub-user space 222, transmitted via the system user space 221 of the rear infotainment system. The source information may, for example, indicate that the gesture recognition results are associated with the first sub-user space 222. Based at least on this source information, the front infotainment system can pair the gesture recognition results from the first sub-user space 222 with voice input from a user located in the left rear cabin area. Furthermore, the gesture recognition results from the first sub-user space 222 can be fused with the voice recognition results of the voice input from the user located in the left rear cabin area to generate a control command to control the left rear reading light 17 to perform an opening operation.

[0052] As described above, the vehicle may be equipped with multiple rear screens 14 mounted on the rear side of the front seats 11. The multiple sub-user spaces of the rear infotainment system may be associated with one of the multiple rear screens 14. The multiple sub-user spaces may independently run services and / or applications associated with their respective rear screens 14. Gesture recognition results acquired by each sub-user space are generated through image recognition of images captured by a camera device 15 integrated into the rear screen 14 associated with that sub-user space.

[0053] exist Figure 1 In the illustrated example, the vehicle is equipped with a first rear screen 141 facing the left rear seat 121 and a second rear screen 142 facing the right rear seat 122. A first sub-user space 222 may be associated with the first rear screen 141, and a second sub-user space 223 may be associated with the second rear screen 142. This facilitates that the first rear screen 141 and the second rear screen 142 can be operated and / or display content independently of each other by the user.

[0054] The first rear screen 141 integrates a first camera device 151, which is configured to capture images of a user located on the left rear seat 121. The second rear screen 142 integrates a second camera device 152, which is configured to capture images of a user located on the right rear seat 122.

[0055] In step S2, the voice dialogue service (SDSService) 225 of the system user space 221 obtains the gesture recognition results of each sub-user space through the domain adapter (see...). Figure 4 This helps to achieve architectural decoupling and reduce system maintenance costs.

[0056] Optionally, the domain adapter can perform unified cleaning, parsing and format normalization of the gesture recognition results of each sub-user space, thereby providing standardized and consistent input data for SDSService, effectively avoiding interaction logic confusion caused by differences in data formats of multiple sub-spaces, and improving the accuracy of the collaborative response of gesture recognition and voice dialogue service 225.

[0057] In an exemplary embodiment according to this application, the target control object may also include vehicle components 10 such as a music player, air conditioner, window, and door lock.

[0058] According to an exemplary embodiment of this application, the voice recognition result obtained by means of the front infotainment system in step S1 also includes additional target control object information obtained by recognizing the user's voice input. In step S3, the front infotainment system can fuse the additional target control object information of the voice recognition result with the target control object information of the matching gesture recognition result to determine the vehicle component 10 to be controlled. This helps to determine the vehicle component 10 to be controlled more accurately and reliably.

[0059] Figure 3 A flowchart illustrating step S3 of a method for controlling vehicle component 10 according to an exemplary embodiment of this application is shown schematically.

[0060] In this embodiment, step S3 of the method for controlling vehicle component 10 includes sub-steps S31, S32, S33, S34, S35, and S36.

[0061] In sub-step S31, the speech recognition result and the gesture recognition result are matched. In particular, the speech recognition result and the gesture recognition result can be matched based on source information and voice region information.

[0062] According to an exemplary embodiment of this application, the method for controlling vehicle component 10 may further include recognizing a user's lip movement characteristics via a rear-seat infotainment system to obtain lip movement recognition results and sending them to a front-seat infotainment system. The front-seat infotainment system may determine the location of the user issuing the voice command based at least on the lip movement recognition results, for matching the voice recognition results with the gesture recognition results.

[0063] In sub-step S32, it is determined whether the successfully matched speech recognition result includes additional target control object information. If the speech recognition result includes additional target control object information, then sub-step S33 is executed; otherwise, sub-step S34 is executed.

[0064] For example, some users may not have a comprehensive understanding of vehicle component 10, and therefore may use vague voice commands when issuing voice commands, such as "turn this on." In this case, the voice recognition result does not include additional target control object information.

[0065] As an example, a user could say "Turn on this light" while simultaneously making an upward gesture. The corresponding voice recognition result could include additional target control object information indicating that the target control object is a light.

[0066] In sub-step S33, it is determined whether the additional target control object information of the speech recognition result conflicts with the target control information of the matching gesture recognition result. If the additional target control object information conflicts with the target control object information, step S34 is executed; if the additional target control object information does not conflict with the target control information, step S35 is executed.

[0067] For example, in noisy environments, speech recognition results may be inaccurate, leading to conflicts between additional target control object information and target control information.

[0068] Furthermore, users may use incorrect names to refer to the vehicle component 10 when issuing voice commands due to a lack of comprehensive understanding of the vehicle component 10. This could also lead to conflicts between the additional target control object information and the target control information.

[0069] In sub-step S34, the target control object is determined solely based on the target control object information from the gesture recognition result, without considering the speech recognition result. Optionally, the target control object is the vehicle component 10 with the highest confidence level that conforms to the target control object information. In other words, the target control object is the vehicle component 10 with the highest confidence level that conforms to the target control object information.

[0070] For example, see Figure 1 The user points upwards, and based on the user's gesture, the target control object can be determined to be either the left rear reading light 17 or the left rear microphone 16 located next to the left rear reading light 17. Based on the target control object information, the confidence level for the left rear reading light 17 as the target control object is 60%, and the confidence level for the left rear microphone 16 located next to the left rear reading light 17 as the target control object is 30%. In this case, the target control object can be determined to be the left rear reading light 17 in sub-step S34.

[0071] In sub-step S35, a target control object is determined based on the target control object information and the additional target control object information. The target control object is a vehicle component 10 that conforms to the target control object information and the additional target control object information.

[0072] In sub-step S36, control commands can be generated based on the target control operation determined in sub-step S34 or S35 and the target control operation of the speech recognition result.

[0073] According to an exemplary embodiment of this application, when the additional target control object information does not conflict with the target control object information, the type of the target control object to be controlled can be determined by the additional target control object information, and the location of the target control object to be controlled can be determined by the target control object information.

[0074] As an example, a user can say "Turn on this light" while simultaneously making an upward gesture. Accordingly, the voice recognition result can include additional target control object information, indicating that the target control object to be controlled is a light. The location of the light to be controlled is determined by matching the target control object information in the gesture recognition result with the voice recognition result.

[0075] In this example, based on the user's upward gesture, the target control object can be further identified as the left rear reading light 17.

[0076] The target control object information from the gesture recognition result can, for example, indicate that the left rear reading light 17 is the target control object with a confidence level of 40%, and the left rear microphone 16 is the target control object with a confidence level of 40%. In this case, the target control object can be determined as the left rear reading light 17 by combining the additional target control object information and the target control object information in sub-step S34.

[0077] According to an exemplary embodiment of this application, if the front infotainment system only obtains the voice recognition result while the rear infotainment system does not recognize a valid gesture, it may be because the user did not make a gesture or the user made a gesture but it was not captured by the camera. In this case, a voice prompt message can be generated, such as "Your gesture was not recognized, please make the gesture again" or "Your gesture was not recognized, please confirm whether to continue."

[0078] According to an exemplary embodiment of this application, if the front infotainment system only obtains the voice recognition result and does not receive the gesture recognition result that matches the voice recognition result, the target control object is determined based on the cabin area where the user who issued the voice input is located.

[0079] As an example, a user in the left rear seat 121 can issue a voice command to "turn on the lights." The front infotainment system only receives the voice recognition result and does not receive a gesture recognition result that matches the voice recognition result. The voice recognition result may include additional target control object information, which indicates that the type of target control object to be controlled is a light. Based on the area of ​​the cabin where the user who issued the voice input is located, the target control object can be further determined to be the left rear reading light 17.

[0080] Figure 4 An architectural block diagram of a vehicle infotainment system according to an exemplary embodiment of this application is schematically shown.

[0081] like Figure 4 As shown, the vehicle infotainment system includes a first controller 21 and a second controller 22.

[0082] The front infotainment system running the first controller 21 and the rear infotainment system running the second controller 22 can run on the same or different operating systems. For example, the front infotainment system can be configured to run on a Linux operating system, and the rear infotainment system can be configured to run on an Android operating system. The rear infotainment system has a system user space 221 and multiple sub-user spaces. The system user space 221 can run the voice dialogue service 225 (SDS Service).

[0083] In this embodiment, the rear-seat infotainment system has two sub-user spaces: a first sub-user space 222 and a second sub-user space 223. These two sub-user spaces are respectively associated with a first rear screen 141 and a second rear screen 142. Each sub-user space operates a voice interaction adapter 224 (SDS Adapter). The voice interaction adapters 224 of different sub-user spaces can independently acquire gesture recognition results associated with their respective cabin areas. The voice interaction adapter 224 of the first sub-user space 222 can acquire gesture recognition results associated with the corresponding left rear cabin area. The voice interaction adapter 224 of the second sub-user space 223 can acquire gesture recognition results associated with the corresponding right rear cabin area.

[0084] The voice dialogue service 225 of the system user space 221 can obtain the gesture recognition results of each sub-user space through the domain adapter. Then, the system user space 221 can send the source information indicating the sub-user space and the gesture recognition results from the sub-user space to the front-end infotainment system.

[0085] The front and rear infotainment systems can exchange structured data via a communication middleware layer 23 that combines Thrift and gRPC architectures. This ensures both the flexibility of data transmission and the high-performance requirements of real-time data interaction, and enables standardized definition and invocation of interfaces between heterogeneous systems.

[0086] Communication between the front and rear infotainment systems can utilize in-vehicle Ethernet as the underlying transmission medium. This facilitates high-bandwidth, low-latency data transmission between the two systems. Ethernet provides a stable transmission channel for Thrift and gRPC, ensuring efficient and reliable cross-system communication.

[0087] The front infotainment system can receive gesture recognition results and corresponding source information from the rear infotainment system.

[0088] On the other hand, the front infotainment system can recognize the user's voice input to obtain voice recognition results containing target control operation information. For example, the vehicle's microphone 16 can detect voice input from a user inside the vehicle. By performing voice recognition on the voice input, a voice recognition result can be obtained. The front infotainment system can also acquire sound zone information associated with the voice recognition result. The sound zone information is used to indicate the location of the user who made the voice input. For example, by utilizing the time difference and intensity difference of sound waves arriving at different microphones 16, the location and direction of the sound source can be determined, thereby generating sound zone information.

[0089] The front-row infotainment system can at least match the speech recognition results and gesture recognition results based on the source information and fuse the matched speech recognition results and gesture recognition results to generate control instructions for controlling the target object to perform target control operations.

[0090] As an example, if the vocal region information of the speech recognition result can match the source information of the gesture recognition result, then the speech recognition result can be matched with the gesture recognition result.

[0091] The front-row infotainment system may have a dialogue engine 211 and a dialogue domain processor 212. The dialogue domain processor 212 may be configured to distribute control commands according to service domains (e.g., microphone 16, multimedia 24, general service domain 25, user interface, etc.). The dialogue domain processor 212 may act as a service sorter for the dialogue engine 211, distributing user voice requests to the corresponding service processing logic.

[0092] According to an exemplary embodiment of this application, the plurality of sub-user spaces of the rear-seat infotainment system can independently determine the location information of a target control object based on gesture input. The target control object location information may indicate the specific location of the vehicle component 10 to be controlled or the area where the vehicle component 10 to be controlled is located.

[0093] Optionally, in different sub-user spaces, the position information of the target control object can be determined independently based on different association relationships according to the gesture input.

[0094] See Figure 5 The user in the left rear seat 121 can give the voice command "close the window" while making a gesture pointing to the upper right.

[0095] The microphone 16 inside the vehicle can detect the user's voice input. Through voice recognition, the front infotainment system can obtain the voice recognition results containing the target control operation information. The target control operation information can instruct the user's target control operation to be "off".

[0096] The first camera device 151 can capture an image of the user located on the left rear seat 121. Through image recognition, the user's gesture can be analyzed to determine that the user's finger is pointing in the upper right direction. The first sub-user space 222 of the rear infotainment system can determine the location information of the target control object based on this gesture. In this example, the location information of the target control object can indicate that the vehicle component 10 to be controlled is located in the roof area. This location information of the target control object can be transmitted to the system user space 221 as a gesture recognition result.

[0097] The rear infotainment system's user space 221 can send gesture recognition results from the first sub-user space 222 to the front infotainment system. After matching the voice recognition results with the gesture recognition results and fusing the matched voice recognition results with the gesture recognition results, the front infotainment system can generate control commands for controlling the sunroof 18 in the roof area to perform a closing operation.

[0098] Figure 6 The illustration shows that a user in the right rear seat 122 can also issue the voice command "close the window" while simultaneously making a gesture pointing to the upper right. Through voice recognition, the front infotainment system can obtain the voice recognition result containing the target control operation information. This target control operation information can instruct the user's target control operation to be "close".

[0099] The second camera device 152 can capture an image of the user located on the right rear seat 122. Through image recognition, the user's gesture can be analyzed to determine that the user's finger is pointing to the upper right. The second sub-user space 223 of the rear infotainment system can determine the location information of the target control object based on this gesture. In this example, the location information of the target control object can indicate that the vehicle component 10 to be controlled is located in the area of ​​the right rear window 19. This location information of the target control object can be transmitted to the system user space 221 as a gesture recognition result.

[0100] although Figure 6 The user's gestures shown Figure 5 The gestures of the users shown are the same, but the target control object position information determined by the first sub-user space 222 and the second sub-user space 223 based on the same gesture input are different.

[0101] The rear infotainment system's user space 221 can send gesture recognition results from the second sub-user space 223 to the front infotainment system. After matching the voice recognition results with the gesture recognition results and fusing the matched voice recognition results with the gesture recognition results, the front infotainment system can generate control commands for controlling the right rear window 19 to perform a closing operation.

[0102] Alternatively, different sub-user spaces may utilize different functions, algorithms, and / or machine learning models to determine the location information of the target control object.

[0103] In one exemplary embodiment according to this application, a first sub-user space 222 uses a first neural network model to determine the position information of a target controlled object based on gesture input, and a second sub-user space 223 uses a second neural network model different from the first neural network model to determine the position information of the target controlled object based on gesture input. This helps to reduce position recognition bias and improve the accuracy of position determination of the target controlled object and the accuracy of gesture interaction.

[0104] The first neural network model and the second neural network model may differ in terms of training data, training strategy, network structure and / or feature extraction method.

[0105] For example, the first neural network model and the second neural network model were trained using different training datasets.

[0106] According to an exemplary embodiment of this application, in step S3 of the method for controlling vehicle component 10, if the front infotainment system receives valid gesture recognition results from two or more sub-user spaces transmitted via system user space 221, all of which have matching valid voice recognition results, and the gesture recognition results and their matching voice recognition results from the two or more sub-user spaces conflict with each other, then a sub-user space with higher priority is selected from the two or more sub-user spaces, and a control command for controlling the target control object to perform a target control operation is generated based on the gesture recognition results from the higher priority sub-user space and the matching voice recognition results.

[0107] If a user in the left rear seat 121 points to the passenger-side ambient light and says "Turn to blue light," while a user in the right rear seat 122 also points to the passenger-side area and says "Turn to yellow light," then both the first sub-user space 222 and the second sub-user space 223 of the rear infotainment system can obtain gesture recognition results containing target control object information. The front infotainment system can obtain gesture recognition results containing target control operation information through voice recognition. After the front infotainment system matches the gesture recognition results from the first sub-user space 222 and the second sub-user space 223 with the voice recognition results respectively, it can determine that the gesture recognition results from the first sub-user space 222 and its matched voice recognition results conflict with the gesture recognition results from the second sub-user space 223 and its matched voice recognition results. In this scenario, if the first sub-user space 222 has a higher priority than the second sub-user space 223, the front infotainment system can generate control commands based on the gesture recognition results from the first sub-user space 222 and the matching voice recognition results. These commands would control the target controlled object to perform a target control operation, such as controlling the ambient lighting in the passenger area to emit blue light. The gesture recognition results from the second sub-user space 223 and their matching voice recognition results will be ignored.

[0108] The priority of a sub-user space can be preset. For example, the priority of the first sub-user space 222 can be set to be higher than that of the second sub-user space 223 by default.

[0109] Optionally, the priority of the sub-user spaces can be set by the user. In particular, the priority of the first sub-user space 222 and the second sub-user space 223 can be set by the user's operation on the front screen 13.

[0110] Optionally, the priority of a sub-user space can be determined based on the identity of the user in the cabin area corresponding to the sub-user space.

[0111] In one exemplary embodiment, the method for controlling vehicle component 10 may further include obtaining the identity information of a user providing gesture input via facial recognition through a rear-seat infotainment system and sending it to a front-seat infotainment system. The front-seat infotainment system may determine whether to generate a control command based on the gesture recognition result associated with the identity information, according to the priority corresponding to the identity information.

[0112] Optionally, the front-row infotainment system may determine whether to generate control commands based on the gesture recognition results associated with the identity information, according to the operation permissions corresponding to the identity information.

[0113] For example, a user might want to unlock the car doors using voice and gesture commands. The rear-seat infotainment system can obtain the user's identity information through facial recognition. Then, the front-seat infotainment system can determine, based on this identity information, that the user does not have the authority to control the door locks. In this case, the front-seat infotainment system will not generate a control command to unlock the doors based on the user's voice and gesture input.

[0114] This application also relates to a computer program product comprising computer program control instructions, wherein, when executed by one or more processors, the computer program control instructions enable the one or more processors to perform a method for controlling a vehicle component 10 according to an exemplary embodiment of this application.

[0115] The method for controlling vehicle component 10 can be implemented by a system for controlling vehicle component 10 according to an exemplary embodiment of this application.

[0116] The computer program control instructions may be stored in a computer-readable storage medium. The computer-readable storage medium may include, for example, high-speed random access memory, and may also include non-volatile memory or volatile solid-state storage devices. The processor may be a central processing unit, or other general-purpose processors, digital signal processors, application-specific integrated circuits, off-the-shelf programmable gate arrays, or other programmable logic devices.

[0117] It should be understood that in describing representative embodiments, the specification may present methods and / or processes as a specific sequence of steps. However, the method or process should not be limited to the specific order of steps described herein, to the extent that the method or process does not depend on the specific order of steps described herein. Other sequences of steps are also possible, as will be understood by those skilled in the art. Therefore, the specific order of steps set forth in the specification should not be construed as a limitation of the claims. Furthermore, the claims relating to the method and / or process should not be limited to the steps performed in the order written, and those skilled in the art will readily understand that these orders may be varied and still remain within the spirit and scope of the embodiments of this application.

[0118] Although specific embodiments of this application are described in detail herein, they are given for illustrative purposes only and should not be construed as limiting the scope of this application. Various substitutions, modifications, and alterations can be conceived without departing from the spirit and scope of this application.

Claims

1. A method for controlling a vehicle component, the method comprising the following steps: -S1, by means of the front infotainment system running on the first controller (21) to recognize the user's voice input, to obtain a voice recognition result containing target control operation information; -S2, by means of the rear-seat infotainment system running on the second controller (22), the user's gesture input is recognized to obtain a gesture recognition result containing target control object information, which is used to indicate the vehicle component (10) to be controlled, wherein, The rear infotainment system sends the gesture recognition results to the front infotainment system; as well as -S3 matches the speech recognition results with the gesture recognition results and merges the matched speech recognition results with the gesture recognition results to generate control instructions for controlling the target object to perform target control operations.

2. The method according to claim 1, wherein, The rear infotainment system has a system user space (221) and multiple sub-user spaces corresponding to different cabin areas. In step S2, the voice dialogue adapter (224) of the sub-user space obtains the gesture recognition result associated with the corresponding cabin area. The system user space (221) sends the source information associated with the sub-user space and the gesture recognition result from the sub-user space to the front infotainment system.

3. The method according to claim 2, wherein, In step S3, the front-row infotainment system matches the speech recognition results and gesture recognition results based at least on the source information and fuses the matched speech recognition results and gesture recognition results to generate control instructions for controlling the target control object to perform target control operations.

4. The method according to claim 2 or 3, wherein, The multiple sub-user spaces of the rear infotainment system independently determine the position information of the target control object based on gesture input.

5. The method according to any one of claims 2-4, wherein, Different sub-user spaces determine the location information of the target controlled object based on gesture input according to different association relationships; and / or Different sub-user spaces utilize different neural network models to determine the position information of the target control object based on gesture input.

6. The method according to any one of claims 2-5, wherein, The vehicle is equipped with multiple rear screens (14) installed on the rear side of the front seats (11) of the vehicle. The multiple sub-user spaces of the rear infotainment system are respectively associated with one of the multiple screens. The gesture recognition result obtained by each sub-user space is generated by image recognition of the image captured by the camera device (15) integrated in the rear screen (14) associated with the sub-user space.

7. The method according to any one of claims 2-6, wherein, In step S3, if the front infotainment system receives valid gesture recognition results from two or more sub-user spaces, and each gesture recognition result has a matching valid voice recognition result, and the gesture recognition results and their matching voice recognition results from the two or more sub-user spaces conflict with each other, then the sub-user space with higher priority is selected from the two or more sub-user spaces, and a control command for controlling the target control object to perform target control operations is generated based on the gesture recognition results from the higher priority sub-user space and the matching voice recognition results.

8. The method according to any one of claims 1-7, wherein, The method includes: if the front infotainment system only obtains the voice recognition result and does not receive the gesture recognition result that matches the voice recognition result, then the target control object is determined based on the cabin area where the user who issued the voice input is located.

9. The method according to any one of claims 1-8, wherein, The voice recognition results obtained in step S1 by means of the front infotainment system also include additional target control object information obtained by recognizing the user's voice input; In step S3, the front infotainment system merges the additional target control object information of the voice recognition result with the target control object information of the matching gesture recognition result in order to determine the vehicle component (10) to be controlled.

10. The method according to claim 9, wherein, The process of integrating the additional target control object information with the target control object information includes the following sub-steps: Determine whether the additional target control object information conflicts with the target control object information; If the additional target control object information conflicts with the target control object information, the target control object is determined solely based on the target control object information, wherein the target control object may optionally be a vehicle component (10) that conforms to the target control object information and has the highest confidence level; and If the additional target control object information does not conflict with the target control object information, the target control object is determined based on the target control object information and the additional target control object information. The target control object is a vehicle component (10) that conforms to the target control object information and the additional target control object information. Optionally, the type of the target control object to be controlled is determined by the additional target control object information, and the position of the target control object to be controlled is determined by the target control object information.

11. The method according to any one of claims 1-10, wherein, The method further includes: using the rear-seat infotainment system to obtain the identity information of the user providing gesture input through facial recognition and sending it to the front-seat infotainment system; In step S3, the front-row infotainment system determines whether to generate control commands based on the gesture recognition results associated with the identity information, according to the operation permissions and / or priorities corresponding to the identity information.

12. The method according to any one of claims 1-11, wherein, The method further includes: using the rear-seat infotainment system to identify the user's lip movement characteristics, obtaining lip movement recognition results and sending them to the front-seat infotainment system; In step S3, the front-row infotainment system determines the location of the user who issued the voice command based at least on the lip movement recognition result, in order to match the voice recognition result with the gesture recognition result.

13. The method according to any one of claims 1-12, wherein, The front and rear infotainment systems run on different operating systems.

14. The method according to any one of claims 1-13, wherein, Structured data is transmitted between the front and rear infotainment systems through a communication middleware layer (23) that combines Thrift and GRPC architectures.

15. The method according to any one of claims 1-14, wherein, The target controlled objects include the vehicle's ambient lighting; and / or Target control operations include adjusting color, brightness, on / off status, and / or flashing mode.

16. A computer program product comprising computer program control instructions, wherein, When the computer program control instructions are executed by one or more processors, the one or more processors are able to perform the method for controlling vehicle components (10) according to any one of claims 1-15.

17. A vehicle, wherein, The vehicles include: The first controller (21) for operating the front-row infotainment system; and The second controller (22) is used to operate the rear-seat infotainment system. The first controller (21) and the second controller (22) are communicatively connected to each other and configured to perform the method for controlling the vehicle component (10) as described in any one of claims 1-15.

Images