Artificial intelligence-based cockpit interaction method, device, equipment and program product

Abstract

The application discloses a cockpit interaction method and device based on artificial intelligence, equipment and program product, comprising: when it is detected that a target passenger is an old person, switching a cockpit interaction mode to an old person mode; acquiring voice information, face image information, body movement information and age range information of the target passenger, inputting the voice information, the face image information, the body movement information and the age range information into a pre-trained old person cockpit interaction intention recognition model to obtain a target interaction intention of the target passenger; and performing interaction control on a target cockpit according to the target interaction intention. The application improves the travel experience of the old person and can be widely applied to the technical field of intelligent cockpits.

Description

Technical Field

[0001] This invention relates to the field of intelligent cockpit technology, and in particular to a cockpit interaction method, device, equipment and program product based on artificial intelligence. Background Technology

[0002] Existing cockpit interaction systems are mainly divided into two categories: one is a general-purpose full-function interaction solution that integrates multiple functions such as navigation, entertainment, vehicle control, and connectivity. The interface elements are complex and the fonts are small. Although the functions are comprehensive, they are not optimized for the elderly, making it very inconvenient for them to operate. The other is a basic simplified solution that only achieves initial simplification by reducing the scope of functions, but it does not solve core pain points such as interface adaptation, voice recognition adaptation, and emergency assistance. It is also not convenient for the elderly to interact with the cockpit.

[0003] In addition, existing technologies often fail to consider the physiological characteristics of the elderly (declining vision, slow reaction speed, and dialect usage habits), and voice interaction only supports standard Mandarin and normal speaking speed, lacking targeted optimization; moreover, the emergency help function requires multiple steps to trigger (such as unlocking the screen - finding the contact - dialing), which cannot meet the needs of the elderly for quick help in emergency situations.

[0004] In summary, existing cockpit interaction systems are incompatible with the cockpit interaction needs of the elderly, affecting their travel experience. These issues urgently need to be addressed. Summary of the Invention

[0005] The purpose of this invention is to at least partially solve one of the technical problems existing in the prior art.

[0006] Therefore, one objective of this invention is to provide an artificial intelligence-based cockpit interaction method that improves the travel experience for the elderly.

[0007] Another objective of this invention is to provide a cockpit interaction device based on artificial intelligence.

[0008] To achieve the above-mentioned technical objectives, the technical solutions adopted in the embodiments of the present invention include: On one hand, embodiments of the present invention provide an artificial intelligence-based cockpit interaction method, comprising the following steps: When the target occupant is detected to be an elderly person, the cockpit interaction mode is switched to elderly mode; The voice information, facial image information, body movement information, and age group information of the target occupant are obtained. The voice information, facial image information, body movement information, and age group information are then input into a pre-trained elderly cockpit interaction intent recognition model to obtain the target occupant's target interaction intent. Interactive control of the target cockpit is performed based on the stated target interaction intent.

[0009] Furthermore, in one embodiment of the present invention, the detection of whether the target occupant is an elderly person is performed through the following steps: When the car door is opened from the outside, the palm print information of the target occupant is obtained through the palm print acquisition device installed on the door handle; The palmprint information is input into a pre-trained occupant age group recognition model to obtain the age group recognition result of the target occupant; Based on the age group identification results, determine whether the target passenger is an elderly person.

[0010] Furthermore, in one embodiment of the present invention, the occupant age group recognition model is trained through the following steps: Obtain palm print samples from multiple testers of different age groups and determine the age group label corresponding to each palm print sample; The palm print samples are input into a pre-constructed CNN-Transformer hybrid neural network to obtain the corresponding predicted age range; A first loss value is determined based on the predicted age group and the age group label; The parameters of the CNN-Transformer hybrid neural network are updated based on the first loss value to obtain the trained occupant age group recognition model; The CNN-Transformer hybrid neural network includes an image segmentation layer, a CNN layer, a Transformer encoding layer, and an MLP regression layer.

[0011] Furthermore, in one embodiment of the present invention, the step of inputting the palmprint sample into a pre-constructed CNN-Transformer hybrid neural network to obtain the corresponding predicted age range specifically includes: The palmprint sample is segmented using the image segmentation layer to obtain multiple palmprint region samples; Multiple palmprint region samples are respectively input into multiple convolutional units of the CNN layer to obtain multiple local palmprint features; The global palmprint features are obtained by performing global feature modeling on the local palmprint features through the Transformer encoding layer. The global palmprint features are mapped to the predicted age range through the MLP regression layer.

[0012] Furthermore, in one embodiment of the present invention, the elderly person's cockpit interaction intent recognition model is trained through the following steps: We acquired voice samples, facial image samples, body movement samples, and age group samples from multiple elderly passengers, and determined the corresponding interaction intent labels through manual annotation. The speech samples, facial image samples, body movement samples, and age group samples are input into a pre-constructed multi-branch convolutional neural network to obtain the corresponding predicted interaction intent. A second loss value is determined based on the predicted interaction intent and the interaction intent label; The parameters of the multi-branch convolutional neural network are updated based on the second loss value to obtain the trained elderly cockpit interaction intention recognition model. The multi-branch convolutional neural network includes a speech branch network, an image branch network, an action branch network, a feature fusion layer, and an output layer.

[0013] Furthermore, in one embodiment of the present invention, the step of inputting the speech samples, the facial image samples, and the body movement samples into a pre-constructed multi-branch convolutional neural network to obtain the corresponding predicted interaction intent specifically includes: The speech sample is input into the speech branch network for semantic feature extraction to obtain a semantic feature vector; The facial image samples are input into the image branch network to extract facial expression features, resulting in facial expression feature vectors. The limb movement samples are input into the action branch network for posture feature extraction to obtain a posture feature vector; The feature fusion layer determines the feature weights of the semantic feature vector, the facial expression feature vector, and the posture feature vector based on the age group samples, and performs feature fusion on the semantic feature vector, the facial expression feature vector, and the posture feature vector based on the feature weights to obtain a fused feature vector. The output layer maps the fused feature vector to the predicted interaction intent.

[0014] Furthermore, in one embodiment of the present invention, the step of interactively controlling the target cockpit according to the target interaction intent specifically includes: Determine the confidence level of the target interaction intent; When the confidence level is greater than or equal to a preset first threshold, a corresponding cockpit control command is generated according to the target interaction intent, and the target cockpit is interactively controlled according to the cockpit control command. When the confidence level is less than the first threshold, a corresponding interaction intent confirmation prompt is generated based on the target interaction intent, and the interaction intent confirmation prompt is broadcast through a voice broadcasting device. If the interaction intent confirmation instruction of the target occupant is received, a corresponding cockpit control command is generated based on the target interaction intent, and the target cockpit is interactively controlled based on the cockpit control command.

[0015] On the other hand, embodiments of the present invention provide an artificial intelligence-based cockpit interaction device, comprising: The interaction mode switching module is used to switch the cockpit interaction mode to elderly mode when the target occupant is detected to be an elderly person. The interaction intent recognition module is used to acquire the voice information, facial image information, body movement information and age group information of the target occupant, and input the voice information, facial image information, body movement information and age group information into a pre-trained elderly cockpit interaction intent recognition model to obtain the target interaction intent of the target occupant. The interactive control module is used to interactively control the target cockpit according to the target's interactive intent.

[0016] On the other hand, embodiments of the present invention provide an electronic device, including: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the above-described artificial intelligence-based cockpit interaction method.

[0017] On the other hand, embodiments of the present invention also provide a computer-readable storage medium storing a processor-executable computer program that, when executed by a processor, implements the aforementioned artificial intelligence-based cockpit interaction method.

[0018] On the other hand, embodiments of the present invention also provide a computer program product, including a computer program that, when executed by a processor, implements the aforementioned artificial intelligence-based cockpit interaction method.

[0019] The advantages and beneficial effects of the present invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention: This invention detects whether a target passenger is elderly when they board the vehicle. If an elderly passenger is detected, the cabin interaction mode is switched to elderly mode. In elderly mode, the system acquires the passenger's voice information, facial image information, body movement information, and age group information. This information is then input into a pre-trained elderly passenger cabin interaction intent recognition model to obtain the passenger's target interaction intent. Based on this intent, the system performs interactive control of the cabin. This invention, by switching the cabin interaction mode to elderly mode upon detecting an elderly passenger and comprehensively recognizing their cabin interaction intent based on voice information, facial image information, body movement information, and age group information, can accurately control the cabin based on the recognized intent, eliminating the need for cumbersome cabin interaction operations and improving the travel experience for the elderly. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the embodiments of the present invention are described below. It should be understood that the drawings described below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 A flowchart illustrating the steps of an artificial intelligence-based cockpit interaction method provided in an embodiment of the present invention; Figure 2 A structural block diagram of an artificial intelligence-based cockpit interaction device provided in an embodiment of the present invention; Figure 3 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of this invention; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this invention as detailed in the appended claims.

[0023] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to limit the invention.

[0024] The AI-based cockpit interaction method provided in this invention can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or in-vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and AI platforms. The server can also be a node server in a blockchain network. The software can be an application that implements the AI-based cockpit interaction method, but is not limited to the above forms.

[0025] This invention can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0026] It should be noted that in various specific embodiments of the present invention, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of the present invention require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to a confirmation page. Only after obtaining the user's separate permission or consent is the necessary user-related data for the normal operation of the embodiments of the present invention acquired.

[0027] Reference Figure 1 This invention provides an artificial intelligence-based cockpit interaction method, which specifically includes the following steps: S101. When the target occupant is detected to be an elderly person, switch the cockpit interaction mode to elderly person mode. S102. Obtain the target occupant's voice information, facial image information, body movement information, and age group information. Input the voice information, facial image information, body movement information, and age group information into a pre-trained elderly cockpit interaction intent recognition model to obtain the target occupant's target interaction intent. S103. Perform interactive control of the target cockpit according to the target's interaction intent.

[0028] In this embodiment of the invention, when an elderly passenger is detected, the cockpit interaction mode is switched to elderly mode. The cockpit interaction intention of the elderly passenger is comprehensively identified based on voice information, facial image information, body movement information and age group information. Thus, the cockpit can be accurately controlled based on the identified cockpit interaction intention, without the need for the elderly to perform cumbersome cockpit interaction operations, thereby improving the travel experience of the elderly.

[0029] As a further optional implementation, the following steps are used to detect whether the target occupant is an elderly person: S201. When the car door is opened from the outside, the palm print information of the target occupant is obtained through the palm print acquisition device installed on the door handle. S202. Input the palm print information into the pre-trained occupant age group recognition model to obtain the age group recognition result of the target occupant; S203. Determine whether the target passenger is an elderly person based on the age group identification results.

[0030] Specifically, in this embodiment of the invention, when a passenger gets into the vehicle, the palm print collection device installed on the door handle is used to obtain the passenger's palm print information. Then, based on the palm print information, the passenger's age group is identified. Based on the age group identification result, it is determined whether the passenger is an elderly person. If so, the cabin interaction mode is switched to elderly person mode.

[0031] As an optional implementation, the occupant age group identification model is trained through the following steps: S301. Obtain palm print samples from multiple testers of different age groups and determine the age group label corresponding to each palm print sample. S302. Input the palm print samples into a pre-built CNN-Transformer hybrid neural network to obtain the corresponding predicted age range; S303. Determine the first loss value based on the predicted age group and age group label; S304. Update the parameters of the CNN-Transformer hybrid neural network based on the first loss value to obtain the trained passenger age group recognition model. The CNN-Transformer hybrid neural network includes an image segmentation layer, a CNN layer, a Transformer encoding layer, and an MLP regression layer.

[0032] Specifically, palm print samples from multiple testers of different age groups are obtained, and the age group label corresponding to each palm print sample is determined. The palm print samples are then input into a pre-built CNN-Transformer hybrid neural network to obtain the corresponding predicted age group. A first loss value is determined based on the predicted age group and the age group label. The parameters of the CNN-Transformer hybrid neural network are updated based on the first loss value to complete one iteration of training. Training is stopped when the number of iterations reaches a preset threshold or the first loss value is lower than the preset threshold, and the trained occupant age group recognition model is obtained.

[0033] It is understood that the embodiments of the present invention identify the age group of passengers based on palm print information, and can quickly switch the cabin interaction mode the moment the passenger opens the car door from the outside, so that the elderly can enjoy cabin interaction services exclusively for the elderly after entering the cabin, further improving the travel experience of the elderly.

[0034] As a further optional implementation, the palmprint samples are input into a pre-built CNN-Transformer hybrid neural network to obtain the corresponding predicted age range, which specifically includes: S3021. Multiple palmprint region samples are obtained by image segmentation of the palmprint samples through an image segmentation layer. S3022. Input multiple palmprint region samples into multiple convolutional units of the CNN layer to obtain multiple local palmprint features; S3023. Global palm print features are obtained by performing global feature modeling on local palm print features through the Transformer encoding layer; S3024. Map global palmprint features to predicted age groups using an MLP regression layer.

[0035] Specifically, the CNN-Transformer hybrid neural network of this invention includes an image segmentation layer, a CNN layer, a Transformer encoding layer, and an MLP regression layer. The image segmentation layer segments the palm print samples to obtain multiple palm print region samples. The multiple palm print region samples are then input into multiple convolutional units of the CNN layer to obtain multiple local palm print features. The Transformer encoding layer then performs global feature modeling on the local palm print features to obtain global palm print features. Finally, the MLP regression layer maps the global palm print features to a predicted age group.

[0036] The occupant age group recognition model trained through the above steps can quickly identify the occupant's age group the moment the occupant opens the car door from the outside, and thus switch the cabin interaction mode to elderly mode when the occupant is identified as an elderly person.

[0037] In some optional embodiments, to ensure seamless switching of the cabin interaction mode when the elderly person is not touching the door handle, a manual switching unit is provided to enable quick switching between elderly mode and standard mode. For example, a shortcut icon for "elderly mode" (with enlarged font and text label) can be provided on the central control screen or rear seat interactive display screen, supporting touch switching. A mode switching button is also provided on the steering wheel for one-click triggering of mode switching. Additionally, an automatic recognition unit can be equipped to automatically identify elderly users through their user profiles (storing elderly user information). Upon entering the vehicle, the elderly person's identity is confirmed via facial recognition (in-vehicle camera), and the system automatically switches to elderly mode. Furthermore, a mode memory unit can be provided, meaning the system remembers the last used mode by default and automatically loads it the next time the vehicle is started, eliminating the need for repeated switching.

[0038] After the cockpit interaction mode is switched to the elderly mode, the cockpit comprehensively collects the elderly passenger's voice information, facial image information, and body movement information. Combined with the recognized age group information of the elderly passenger (further subdivision of the elderly age group), the cockpit identifies the elderly passenger's true interaction intention and thus performs cockpit interaction control. The elderly passenger does not need to perform a complete interaction operation, which greatly reduces the interaction difficulty for the elderly passenger.

[0039] In some optional embodiments, after switching to the elderly mode, the visual and operational logic of the cockpit interaction system is optimized, retaining only the three core functions (navigation, telephone, and air conditioning), while hiding non-essential functions such as entertainment, connectivity, and vehicle control settings. The interface displays only three function icons without any other redundant elements. At the same time, the font on the central control screen is automatically enlarged by 60% (based on the standard mode font size), and the icon size is enlarged by 80%, using a high-contrast color scheme (black text on a white background with red accent color) to improve visibility. In addition, the operation steps for each core function are simplified. For example, navigation only requires "enter destination → start navigation," and air conditioning only requires "adjust temperature → confirm mode," without multi-level menu jumps, making it easier for the elderly to perform manual interaction operations on their own.

[0040] As an optional implementation, the cockpit interaction intent recognition model for the elderly is trained through the following steps: S401. Obtain voice samples, facial image samples, body movement samples, and age group samples from multiple elderly passengers, and determine the corresponding interaction intent labels through manual annotation. S402. Input the speech samples, facial image samples, body movement samples and age group samples into a pre-built multi-branch convolutional neural network to obtain the corresponding predicted interaction intent. S403. Determine the second loss value based on the predicted interaction intent and the interaction intent label; S404. Update the parameters of the multi-branch convolutional neural network according to the second loss value to obtain the trained cockpit interaction intention recognition model for the elderly. The multi-branch convolutional neural network includes a speech branch network, an image branch network, an action branch network, a feature fusion layer, and an output layer.

[0041] Specifically, the elderly passenger cabin interaction intent recognition model of this invention is trained based on a multi-branch convolutional neural network. It extracts multi-dimensional features such as semantics, expression, and posture from multi-dimensional data including speech, facial images, and body movements. The feature weights of each feature are adaptively determined by combining the passenger's age group, thereby obtaining fused features to recognize the interaction intent of elderly passengers. Specifically, it acquires speech samples, facial image samples, body movement samples, and age group samples from multiple elderly passengers, and determines corresponding interaction intent labels through manual annotation. The speech samples, facial image samples, body movement samples, and age group samples are input into a pre-constructed multi-branch convolutional neural network to obtain the corresponding predicted interaction intent. A second loss value is determined based on the predicted interaction intent and the interaction intent label. The parameters of the multi-branch convolutional neural network are updated based on the second loss value to complete one iteration of training. Training stops when the number of iterations reaches a preset threshold or the second loss value falls below the preset threshold, thus obtaining the trained elderly passenger cabin interaction intent recognition model.

[0042] As a further optional implementation, speech samples, facial image samples, and body movement samples are input into a pre-constructed multi-branch convolutional neural network to obtain the corresponding predicted interaction intent, which specifically includes: S4021. Input the speech sample into the speech branch network to extract semantic features and obtain the semantic feature vector; S4022. Input facial image samples into an image branching network to extract facial expression features and obtain facial expression feature vectors; S4023. Input the limb motion samples into the motion branch network to extract posture features and obtain the posture feature vector. S4024. The feature fusion layer determines the feature weights of semantic feature vector, facial expression feature vector and posture feature vector based on the age group samples, and performs feature fusion on the semantic feature vector, facial expression feature vector and posture feature vector based on the feature weights to obtain the fused feature vector. S4025. The fused feature vectors are mapped to predicted interaction intentions through the output layer.

[0043] Specifically, this embodiment of the invention designs a cockpit interaction intent recognition model for elderly passengers. While considering multi-dimensional data such as voice information, facial image information, and body movement information, it also inputs the passenger's age group information into the model to adaptively adjust the weights of each dimension of features. Since elderly people may experience hearing loss or unclear speech, body movements and facial expressions more accurately reflect intent as age increases. Therefore, the feature fusion layer of this embodiment adapts the determined feature weights of semantic feature vectors, facial expression feature vectors, and posture feature vectors based on the passenger's age group information. Then, it fuses the semantic feature vectors, facial expression feature vectors, and posture feature vectors according to the feature weights to obtain a fused feature vector. For example, when the elderly passenger is 60-65 years old, the feature weights of the semantic feature vector, facial expression feature vector, and posture feature vector are 0.3, 0.3, and 0.4, respectively. When the elderly passenger is 75-80 years old, the feature weights of the semantic feature vector, facial expression feature vector, and posture feature vector are adaptively adjusted to 0.1, 0.4, and 0.5, thereby more accurately recognizing the cockpit interaction intent of elderly passengers.

[0044] It should be noted that the speech branch network in this embodiment of the invention supports slow speech and dialect recognition, improving the success rate of voice interaction. It has a built-in slow speech recognition model, which supports the recognition of slow speech commands at a rate of 50-70 words per minute (the standard speech rate is 120-150 words per minute). It integrates eight mainstream dialect recognition models (Cantonese, Sichuan-Chongqing dialect, Jiangsu-Zhejiang dialect, Northeastern dialect, Henan dialect, Shandong dialect, Hunan dialect, and Fujian dialect), and supports the recognition of mixed dialect and Mandarin commands.

[0045] It can be recognized that in the elderly mode, by acquiring the target occupant's voice information, facial image information, and body movement information, and combining this with the age group information identified when the target occupant opens the car door, and inputting it into the elderly cockpit interaction intent recognition model trained through the above steps, the target occupant's target interaction intent can be obtained.

[0046] As a further optional implementation, interactive control of the target cockpit is performed according to the target's interaction intent, specifically including: S1031. Determine the confidence level of the target interaction intent; S1032. When the confidence level is greater than or equal to the preset first threshold, generate the corresponding cockpit control command according to the target interaction intention, and perform interactive control on the target cockpit according to the cockpit control command. S1033. When the confidence level is less than the first threshold, generate a corresponding interaction intent confirmation prompt based on the target interaction intent, and broadcast the interaction intent confirmation prompt through the voice broadcasting device. If the target occupant's interaction intent confirmation instruction is received, generate a corresponding cockpit control command based on the target interaction intent, and perform interactive control on the target cockpit based on the cockpit control command.

[0047] Specifically, when the target occupant's target interaction intention is identified, the confidence level of the target interaction intention is output simultaneously. If the confidence level is greater than or equal to a first threshold (e.g., 80%), a corresponding cockpit control command is generated based on the target interaction intention, and the target cockpit is interactively controlled according to the cockpit control command. If the confidence level is less than the first threshold, a corresponding interaction intention confirmation prompt is generated based on the target interaction intention, and the interaction intention confirmation prompt is broadcast through a voice broadcast device, thereby confirming with the target occupant whether the target interaction intention matches the target occupant's true intention.

[0048] In some optional embodiments, the cabin is also equipped with an emergency call module, which can enable quick assistance via physical buttons. For example, a red "One-Click Call for Children" physical button is installed on the left or right side of the seat. Pressing it triggers an emergency call, automatically dialing 1-3 preset children's contact numbers (in sequence). If the first contact does not answer, it automatically switches to the next one. The call is automatically put into hands-free mode. At the same time, the real-time location and driving status (stationary / moving) of the vehicle are pushed to the children's mobile APP, making it easy for the children to locate the vehicle quickly.

[0049] The method steps of the embodiments of the present invention have been described above. It can be understood that when an elderly passenger is detected, the embodiment of the present invention switches the cabin interaction mode to the elderly mode. Based on voice information, facial image information, body movement information, and age group information, the cockpit interaction intention of the elderly passenger is comprehensively identified. Therefore, the cockpit can be accurately controlled based on the identified cockpit interaction intention, without the need for the elderly to perform cumbersome cockpit interaction operations, thus improving the travel experience of the elderly.

[0050] Reference Figure 2 This invention provides an artificial intelligence-based cockpit interaction device, comprising: The interaction mode switching module is used to switch the cockpit interaction mode to elderly mode when the target occupant is detected to be an elderly person. The interaction intent recognition module is used to acquire the target occupant's voice information, facial image information, body movement information, and age group information. The voice information, facial image information, body movement information, and age group information are input into a pre-trained elderly cockpit interaction intent recognition model to obtain the target occupant's target interaction intent. The interactive control module is used to interactively control the target cockpit according to the target's interactive intent.

[0051] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0052] Reference Figure 3 This invention provides an electronic device, comprising: At least one processor; At least one memory for storing at least one program; When the above-mentioned at least one program is executed by the above-mentioned at least one processor, the above-mentioned at least one processor implements the above-mentioned artificial intelligence-based cockpit interaction method.

[0053] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0054] This invention also provides a computer-readable storage medium storing a processor-executable computer program that, when executed by a processor, implements the aforementioned artificial intelligence-based cockpit interaction method.

[0055] A computer-readable storage medium according to an embodiment of the present invention can execute an artificial intelligence-based cockpit interaction method provided in an embodiment of the present invention, and can execute any combination of implementation steps of the method embodiment, possessing the corresponding functions and beneficial effects of the method.

[0056] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned artificial intelligence-based cockpit interaction method.

[0057] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0058] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0059] The embodiments described in this invention are for the purpose of more clearly illustrating the technical solutions of the embodiments of this invention, and do not constitute a limitation on the technical solutions provided by the embodiments of this invention. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this invention are also applicable to similar technical problems.

[0060] The terms "first," "second," "third," "fourth," etc. (if present) in the specification and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0061] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the aforementioned blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and sub-operations described as part of a larger operation are executed independently.

[0062] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the aforementioned functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0063] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0064] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0065] More specific examples (a non-exhaustive list) of computer-readable media include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the aforementioned program can be printed, because the aforementioned program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0066] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0067] In the foregoing description of this specification, references to terms such as "one embodiment," "another embodiment," or "some embodiments" indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of the present invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0068] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0069] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of the present invention.

Claims

1. A cockpit interaction method based on artificial intelligence, characterized in that, Includes the following steps: When the target occupant is detected to be an elderly person, the cockpit interaction mode is switched to elderly mode; The voice information, facial image information, body movement information, and age group information of the target occupant are obtained. The voice information, facial image information, body movement information, and age group information are then input into a pre-trained elderly cockpit interaction intent recognition model to obtain the target occupant's target interaction intent. Interactive control of the target cockpit is performed based on the stated target interaction intent.

2. The cockpit interaction method based on artificial intelligence according to claim 1, characterized in that, The following steps are used to determine whether the target occupant is an elderly person: When the car door is opened from the outside, the palm print information of the target occupant is obtained through the palm print acquisition device installed on the door handle; The palmprint information is input into a pre-trained occupant age group recognition model to obtain the age group recognition result of the target occupant; Based on the age group identification results, determine whether the target passenger is an elderly person.

3. The cockpit interaction method based on artificial intelligence according to claim 2, characterized in that, The occupant age group identification model is trained through the following steps: Obtain palm print samples from multiple testers of different age groups and determine the age group label corresponding to each palm print sample; The palm print samples are input into a pre-constructed CNN-Transformer hybrid neural network to obtain the corresponding predicted age range; A first loss value is determined based on the predicted age group and the age group label; The parameters of the CNN-Transformer hybrid neural network are updated based on the first loss value to obtain the trained occupant age group recognition model; The CNN-Transformer hybrid neural network includes an image segmentation layer, a CNN layer, a Transformer encoding layer, and an MLP regression layer.

4. The cockpit interaction method based on artificial intelligence according to claim 3, characterized in that, The step of inputting the palmprint sample into a pre-constructed CNN-Transformer hybrid neural network to obtain the corresponding predicted age range specifically includes: The palmprint sample is segmented using the image segmentation layer to obtain multiple palmprint region samples; Multiple palmprint region samples are respectively input into multiple convolutional units of the CNN layer to obtain multiple local palmprint features; The global palmprint features are obtained by performing global feature modeling on the local palmprint features through the Transformer encoding layer. The global palmprint features are mapped to the predicted age range through the MLP regression layer.

5. The cockpit interaction method based on artificial intelligence according to claim 1, characterized in that, The elderly person's cockpit interaction intent recognition model is trained through the following steps: We acquired voice samples, facial image samples, body movement samples, and age group samples from multiple elderly passengers, and determined the corresponding interaction intent labels through manual annotation. The speech samples, facial image samples, body movement samples, and age group samples are input into a pre-constructed multi-branch convolutional neural network to obtain the corresponding predicted interaction intent. A second loss value is determined based on the predicted interaction intent and the interaction intent label; The parameters of the multi-branch convolutional neural network are updated based on the second loss value to obtain the trained elderly cockpit interaction intention recognition model. The multi-branch convolutional neural network includes a speech branch network, an image branch network, an action branch network, a feature fusion layer, and an output layer.

6. The cockpit interaction method based on artificial intelligence according to claim 5, characterized in that, The step of inputting the speech samples, facial image samples, and body movement samples into a pre-constructed multi-branch convolutional neural network to obtain the corresponding predicted interaction intent specifically includes: The speech sample is input into the speech branch network for semantic feature extraction to obtain a semantic feature vector; The facial image samples are input into the image branch network to extract facial expression features, resulting in facial expression feature vectors. The limb movement samples are input into the action branch network for posture feature extraction to obtain a posture feature vector; The feature fusion layer determines the feature weights of the semantic feature vector, the facial expression feature vector, and the posture feature vector based on the age group samples, and performs feature fusion on the semantic feature vector, the facial expression feature vector, and the posture feature vector based on the feature weights to obtain a fused feature vector. The output layer maps the fused feature vector to the predicted interaction intent.

7. A cockpit interaction method based on artificial intelligence according to any one of claims 1 to 6, characterized in that, The step of interactively controlling the target cockpit according to the target interaction intent specifically includes: Determine the confidence level of the target interaction intent; When the confidence level is greater than or equal to a preset first threshold, a corresponding cockpit control command is generated according to the target interaction intent, and the target cockpit is interactively controlled according to the cockpit control command. When the confidence level is less than the first threshold, a corresponding interaction intent confirmation prompt is generated based on the target interaction intent, and the interaction intent confirmation prompt is broadcast through a voice broadcasting device. If the interaction intent confirmation instruction of the target occupant is received, a corresponding cockpit control command is generated based on the target interaction intent, and the target cockpit is interactively controlled based on the cockpit control command.

8. A cockpit interaction device based on artificial intelligence, characterized in that, include: The interaction mode switching module is used to switch the cockpit interaction mode to elderly mode when the target occupant is detected to be an elderly person. The interaction intent recognition module is used to acquire the voice information, facial image information, body movement information and age group information of the target occupant, and input the voice information, facial image information, body movement information and age group information into a pre-trained elderly cockpit interaction intent recognition model to obtain the target interaction intent of the target occupant. The interactive control module is used to interactively control the target cockpit according to the target's interactive intent.

9. An electronic device, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements an artificial intelligence-based cockpit interaction method as described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements an artificial intelligence-based cockpit interaction method as described in any one of claims 1 to 7.

Images