Vehicle-mounted man-machine interaction method, device and equipment and storage medium

Abstract

The invention provides a vehicle-mounted man-machine interaction method, device and equipment and a storage medium, and the method comprises the steps: shooting environment images through high-definition cameras installed at the head and the tail of a train, and collecting environment data through an integrated sensor; aligning the environment image and the environment data in a shared space at the front end to generate perception information; in the back end, using the first artificial intelligence model to judge the current operation complex condition, calling the matched computing resource according to the current operation complex condition, and using the computing resource to execute the second artificial intelligence model to perform fault prediction based on the perception information to obtain an early warning result; a man-machine interaction mode is set in a vehicle-mounted interface, a detection unit for displaying an early warning result, a blind area detection unit for displaying an environment image and units for controlling train driving are deployed, and the display sequence of the units in the vehicle-mounted interface is adjusted in real time by using an online learning algorithm. A personalized layout adjustment mode is provided, and the flexibility is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of human-computer interaction, and in particular to a vehicle-mounted human-computer interaction method, device, equipment and storage medium. BACKGROUND

[0002] With the development of urban rail transit systems, the intelligent level of trains is getting higher and higher.

[0003] In related technologies, the existing train-mounted system usually focuses on the overall control and monitoring of train operation, such as the running state, speed, scheduling, etc. of the train. Although the driver can master part of the running conditions of the train, the above technical means currently still have the following defects: the driver has a visual dead angle that cannot be directly viewed during driving, especially the tail situation; the system fails to integrate data from multiple sensors and image information to comprehensively detect or warn potential safety hazards; and the isolation of each information of the traditional vehicle-mounted system leads to multi-interface operation, usually using static display interfaces and physical buttons, lacking intelligent and personalized interface adjustment functions, and being unable to automatically optimize according to the environment or driver needs.

[0004] In summary of the above analysis of the development status of the technical field, the existing technology lacks a scheme of deploying high-definition cameras at the front and rear of the train, integrating multi-sensor data and image data for alignment to perform fault prediction, and using an online learning algorithm to automatically adjust the interface layout. SUMMARY

[0005] The purpose of the present application is to provide a vehicle-mounted human-computer interaction method, device, equipment and storage medium, aiming to solve the above problems in the prior art.

[0006] According to a first aspect of an embodiment of the present application, a vehicle-mounted human-computer interaction method is provided, comprising:

[0007] capturing environment images through high-definition cameras installed at the front and rear of the train, and collecting environment data through integrated sensors;

[0008] aligning the environment images and the environment data in a shared space in the front end to generate perception information;

[0009] using a first artificial intelligence model in the back end to judge the current running complexity, calling matching computing resources according to the current running complexity, using the computing resources to execute a second artificial intelligence model based on the perception information to perform fault prediction, and obtaining a warning result;

[0010] setting a human-computer interaction mode in the vehicle-mounted interface, and deploying a detection unit for displaying the warning result, a blind area detection unit for displaying the environment images, and each unit for controlling the driving of the train, and using an online learning algorithm to adjust the display order of each unit in the vehicle-mounted interface in real time.

[0011] According to a second aspect of the embodiments of the present application, a vehicle-mounted human-computer interaction device is provided, comprising:

[0012] An initial acquisition module is configured to capture environment images by high-definition cameras installed at the head and tail of a train and collect environment data by integrated sensors;

[0013] A front-end fusion module is configured to align the environment images and the environment data in a shared space in the front end to generate perception information;

[0014] A back-end execution module is configured to determine a current running complexity in the back end using a first artificial intelligence model, call matched computing resources according to the current running complexity, perform a second artificial intelligence model based on the perception information using the computing resources to make fault prediction, and obtain an early warning result;

[0015] An adaptive adjustment module is configured to set a human-computer interaction mode in a vehicle-mounted interface, deploy a detection unit for displaying the early warning result, a blind area detection unit for displaying the environment images, and units for controlling train driving, and use an online learning algorithm to adjust the display order of each unit in the vehicle-mounted interface in real time.

[0016] According to a third aspect of the embodiments of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, which, when executed by the processor, implements the steps of the vehicle-mounted human-computer interaction method provided in the first aspect of the present application.

[0017] According to a fourth aspect of the embodiments of the present application, a computer-readable storage medium is provided, which stores an implementation program of information transmission, which, when executed by a processor, implements the steps of the vehicle-mounted human-computer interaction method provided in the first aspect of the present application.

[0018] The technical scheme provided by the embodiments of the present application has the following beneficial effects: deploying high-definition cameras at the head and tail of a train enables a driver to timely grasp the protection information of the tail; aligning integrated multi-sensor data and image data for fault prediction, in which image data provides high-resolution texture and voice information, and radar sensor data provides accurate geometric structure, complementarily fusing the two to improve prediction accuracy; multi-functional area division enables a driver to view or operate without fear of frequent interface switching to reduce use complexity, and provides a personalized layout adjustment mode to improve flexibility.

[0019] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the one or more embodiments of the present specification or the prior art, the drawings needed to be used in the embodiment or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description only represent some of the embodiments described in the present specification, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0021] Figure 1 is a flowchart of the vehicle-mounted man-machine interaction method of the embodiment of the present application;

[0022] Figure 2 is a schematic diagram of the vehicle-mounted man-machine interaction device of the embodiment of the present application;

[0023] Figure 3 is a schematic diagram of the electronic device of the embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to make the person skilled in the art better understand the technical solutions in the one or more embodiments of the present specification, the technical solutions in the one or more embodiments of the present specification will be described clearly and completely in the following with reference to the drawings in the one or more embodiments of the present specification. Obviously, the described embodiments only represent some of the embodiments described in the present specification, rather than all the embodiments. Based on the one or more embodiments of the present specification, all other embodiments obtained by those skilled in the art without creative labor should belong to the protection scope of the present document.

[0025] Method embodiment

[0026] According to the embodiment of the present application, a vehicle-mounted man-machine interaction method is provided, Figure 1 is a flowchart of the vehicle-mounted man-machine interaction method of the embodiment of the present application, as Figure 1 shown, the vehicle-mounted man-machine interaction method according to the embodiment of the present application specifically includes:

[0027] In step S110, the environment image is captured by the high-definition camera installed at the head and tail of the train, and the environment data is collected by the integrated sensor, specifically including:

[0028] The brightness and contrast of the high-definition camera are adaptively adjusted according to the weather condition. If the weather is not good or other similar conditions, such as driving in a tunnel or at night, the system automatically adjusts the brightness and contrast to ensure that the driver can still clearly see the surrounding situation in low light weather conditions;

[0029] The environment data is collected by the integrated sensor including laser radar and millimeter wave radar;

[0030] In the embodiment of the present application, the integrated sensor also includes sensors for measuring battery power, speed, temperature, etc. to provide more comprehensive information for real-time monitoring of the running state of the train. Preferably, a normal threshold range can be set to monitor whether an abnormality occurs, for example, if the voltage of the on-board battery exceeds the preset range, the system will timely remind. The above sensors are not easy to align with the environment image, so if such data information needs to be used when fault prediction is performed, it is fused in the form of data splicing.

[0031] In step S120, the environment image is aligned with the environment data in the shared space in the front end to generate perception information, specifically including:

[0032] In the traditional data processing mode, sensor data is often processed and calculated independently, and after all the processing, the fusion is performed in the back end. This mode is easy to cause delay and reduce the accuracy of information, so the front-end fusion technology is set in the embodiment of the present application.

[0033] The environment image photographed by the high-definition camera and the environment data collected by the integrated sensor are transmitted to the shared space in real time, the environment image is preprocessed to obtain image information, and the environment data is preprocessed to obtain data information; that is, data preprocessing and fusion are performed almost simultaneously with data collection;

[0034] The data information is converted from the radar coordinate system to the image coordinate system through the calibration matrix to obtain a point cloud projection result, a matrix with the same size as the image information is created, the pixel coordinates of the point cloud projection result are sequentially filled in the corresponding positions of the matrix with the same size, and the missing values are filled using a bilateral filtering method to obtain a depth information matrix.

[0035] The depth information matrix and the matrix of the image information are merged into perception information, and the perception information is an enhanced image with depth information, which is more likely to highlight obstacles in a complex environment to achieve more accurate fault prediction.

[0036] In step S130, a first artificial intelligence model is used in the back end to judge the current running complexity, a matching computing resource is called according to the current running complexity, a second artificial intelligence model is executed using the computing resource based on the perception information to perform fault prediction, and a warning result is obtained, specifically including:

[0037] The scene change rate of the continuous frames photographed by the high-definition camera and the data throughput processed by the sensor, such as point cloud density, are extracted as judgment index features, a lightweight convolution model is used as the first artificial intelligence model, the judgment index features are input into the first artificial intelligence model, and the current running complexity is output.

[0038] By acquiring computing resources from the dynamically allocatable units through the current running complex situation, the use of computing resources can be dynamically adjusted, the more complex the situation, the more computing resources allocated, and in a complex environment, the ability to maintain efficient real-time data processing is maintained, while in a simple scenario, unnecessary computing burden is reduced to optimize energy efficiency and extend the service life of the device.

[0039] Using a long short-term memory network as a second artificial intelligence model to output an early warning result;

[0040] According to the early warning result, the safety hazard level is evaluated, and the alarm is triggered according to the safety hazard level to inform the driver; preferably, the driver is provided with an operable fault solution or emergency measure through a large language model.

[0041] In step S140, a man-machine interaction mode is set in the vehicle-mounted interface, and detection units for displaying early warning results, blind area detection units for displaying environment images, and units for controlling train driving are deployed, and an online learning algorithm is used to adjust the display order of each unit in the vehicle-mounted interface in real time, specifically including:

[0042] The vehicle-mounted interface system is called THMI, which sets a man-machine interaction mode including a touch screen, voice recognition and gesture control, greatly improving the operation convenience of the driver; the units for controlling train driving are other functional units in the vehicle-mounted screen under normal circumstances, such as navigation, multimedia, air conditioning and light control, etc.

[0043] According to the preset operation rule library, the operation deviation of the driver using the man-machine interaction mode is corrected in real time.

[0044] The context tiger algorithm is used to adjust the display order in real time, the operation habit of the driver and the driving environment scene extracted from the environment image are taken as the state, the sorting adjustment of the display order is taken as the action, and the score of the operation completion time is taken as the reward.

[0045] Preferably, in order to ensure the continuous performance of THMI, the embodiment of the present application supports OTA remote update background function, the system can automatically detect new version and update in the background without manual intervention of the driver, the latest optimization control algorithm is obtained in time through remote update, and the long-term stability and reliability of the system are improved.

[0046] To sum up, aiming at the existing problems, the vehicle-mounted man-machine interaction method of the present application deploys high-definition cameras at the train head and tail, and transmits the shooting information to the vehicle-mounted display interface in real time, so that the driver can timely master the protection information of the train tail; the integrated multi-sensor data and image data are aligned for fault prediction, the data is preprocessed and fused in the front end in time after being obtained in the front end, and the waiting time for unified processing in the back end is avoided; during data preprocessing and alignment processing, the image data provides high-resolution texture and voice information, and the integrated sensor data of the laser radar and millimeter wave radar provides accurate geometric structure, which are complementary and fused to improve the prediction accuracy and timely explore potential risks to improve the safety and reliability of the train; the man-machine interaction mode including touch screen, voice recognition and gesture control is set, which greatly improves the convenience of the driver's operation; the multi-functional area division enables the driver to view or operate without fear of frequent interface switching to reduce the use complexity; and the personalized layout adjustment mode is provided to improve flexibility.

[0047] Device embodiment

[0048] According to the embodiment of the present application, a vehicle-mounted man-machine interaction device is provided, Figure 2 is a schematic diagram of the vehicle-mounted man-machine interaction device of the embodiment of the present application, as Figure 2 shown, the vehicle-mounted man-machine interaction device according to the embodiment of the present application specifically comprises:

[0049] The initial acquisition module 20 is used for shooting environment images through high-definition cameras installed at the train head and tail, and collecting environment data through integrated sensors, and specifically used for:

[0050] Adaptively adjusting the brightness and contrast of the high-definition camera shooting according to the weather condition;

[0051] Collecting environment data through integrated sensors including laser radar and millimeter wave radar.

[0052] The front-end fusion module 22 is used for aligning the environment images and environment data in the shared space in the front end to generate perception information, and specifically used for:

[0053] Real-time transmitting the environment images shot by the high-definition cameras and the environment data collected by the integrated sensors into the shared space, pre-processing the environment images to obtain image information, and pre-processing the environment data to obtain data information;

[0054] Converting the data information from the radar coordinate system to the image coordinate system through a calibration matrix to obtain a point cloud projection result, creating a same-size matrix as the image information, sequentially filling in the pixel coordinates of the point cloud projection result at the corresponding positions of the same-size matrix, and using a bilateral filtering method to fill in the missing values to obtain a depth information matrix;

[0055] The depth information matrix is combined with the matrix of image information into perception information.

[0056] The backend execution module 24 is configured to determine a current operation complexity using a first artificial intelligence model in the backend, call a matched computing resource according to the current operation complexity, perform a second artificial intelligence model using the computing resource based on the perception information to obtain a pre-warning result, and specifically configured to:

[0057] The scene change rate of continuous frames captured by a high-definition camera and the data throughput processed by a sensor are extracted as a judgment index feature, a lightweight convolution model is used as the first artificial intelligence model, the judgment index feature is input into the first artificial intelligence model, and the current operation complexity is output.

[0058] The computing resource is obtained from a dynamically allocatable unit through the current operation complexity.

[0059] The long short-term memory network is used as the second artificial intelligence model to output the pre-warning result.

[0060] The safety hazard level is evaluated according to the pre-warning result, and the driver is informed of the alarm according to the safety hazard level.

[0061] The adaptive adjustment module 26 is configured to set a human-computer interaction mode in a vehicle-mounted interface, deploy a detection unit for displaying the pre-warning result, a blind area detection unit for displaying an environment image, and each unit for controlling train driving, and use an online learning algorithm to adjust the display order of each unit in the vehicle-mounted interface in real time, and specifically configured to:

[0062] The human-computer interaction mode including a touch screen, voice recognition, and gesture control is set.

[0063] The operation deviation of the driver using the human-computer interaction mode is corrected in real time according to a preset operation rule library.

[0064] The display order is adjusted in real time using a contextual tiger machine algorithm, the operation habit of the driver and the driving environment scene extracted from the environment image are used as states, the sorting adjustment of the display order is used as an action, and the score of the operation completion time is used as a reward.

[0065] To sum up, aiming at the existing problems, the vehicle-mounted man-machine interaction device is deployed with high-definition cameras at the train head and tail, and the shooting information is transmitted to the vehicle-mounted display interface in real time, so that the driver can timely master the protection information of the train tail; the integrated multi-sensor data and image data are aligned for fault prediction, the data is preprocessed and fused in the front end in time after being acquired in the front end, and the waiting time of unified processing in the rear end is avoided; during the data preprocessing and alignment processing, the image data provides high-resolution texture and voice information, and the integrated sensor data of the laser radar and the millimeter wave radar provides accurate geometric structure, the two are complementary and fused to improve the prediction accuracy, potential risks can be timely discovered to improve the safety and reliability of the train; the man-machine interaction mode including the touch screen, voice recognition and gesture control is set, the convenience of the driver operation is greatly improved; the multi-functional area division enables the driver to view or operate without fear of frequent interface switching to reduce the use complexity; and the personalized layout adjustment mode is provided to improve the flexibility.

[0066] Electronic device embodiment

[0067] Figure 3 is a schematic diagram of an electronic device according to an embodiment of the application. The electronic device 300 can include at least one processor 310 and a memory 320. The processor 310 can execute instructions stored in the memory 320. The processor 310 is communicatively connected to the memory 320 through a data bus. In addition to the memory 320, the processor 310 can also be communicatively connected to an input device 330, an output device 340, and a communication device 350 through the data bus.

[0068] The processor 310 can be any conventional processor, such as a commercially available CPU. The processor can also include a graphic processing unit (GPU), a field programmable gate array (FPGA), a system on chip (SOC), an application specific integrated circuit (ASIC), or a combination thereof.

[0069] The memory 320 can be implemented by any type of volatile or non-volatile storage devices or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic storage, a flash memory, a magnetic disk, or an optical disk.

[0070] In the embodiments of the present disclosure, the executable instructions are stored in the memory 320, and the processor 310 can read the executable instructions from the memory 320 and execute the instructions to implement all or part of the steps of the vehicle-mounted human-computer interaction method of any of the above example embodiments.

[0071] Embodiment of computer-readable storage medium

[0072] In addition to the above method and device, the example embodiments of the present disclosure can also be a computer program product or a computer-readable storage medium storing the computer program product, the computer program product including computer program instructions executable by a processor to implement all or part of the steps described in the vehicle-mounted human-computer interaction method of any of the above example embodiments.

[0073] The computer program product can be written in any combination of one or more programming languages to perform the operations of the embodiments of the present application, including object-oriented programming languages such as Java, C++, and conventional procedural programming languages such as "C" language or similar programming languages and scripting languages (e.g., Python). The program code can be executed entirely on a user computing device, partially on a user device, as an independent software package, partially on a user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0074] The computer-readable storage medium can employ any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium may, for example, include but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the above. More specific examples of readable storage media include static random access memory (SRAM) with one or more conductive lines electrically connected, electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk, or any suitable combination of the above.

[0075] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the above embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A vehicle-mounted human-computer interaction method, characterized in that, include: High-definition cameras installed at the front and rear of the train capture environmental images, and integrated sensors collect environmental data. Align the environmental image and the environmental data in the shared space at the front end to generate perceptual information; In the backend, a first artificial intelligence model is used to determine the current operational complexity. Based on the current operational complexity, matching computing resources are called. Based on the perception information, the computing resources are used to execute a second artificial intelligence model to predict faults and obtain early warning results. A human-computer interaction method is set in the vehicle interface, and a detection unit that displays the warning results, a blind spot detection unit that displays the environmental image, and various units that control train driving are deployed. An online learning algorithm is used to adjust the display order of each unit in the vehicle interface in real time.

2. The method according to claim 1, characterized in that, The process of capturing environmental images using high-definition cameras installed at the front and rear of the train, and collecting environmental data through integrated sensors, specifically includes: The brightness and contrast of the high-definition camera are adaptively adjusted according to weather conditions. Environmental data is collected through an integrated sensor system that includes lidar and millimeter-wave radar.

3. The method according to claim 1, characterized in that, The step of aligning the environmental image and the environmental data in the shared space of the front end to generate perception information specifically includes: High-definition camera-captured environmental images and integrated sensor-collected environmental data are transmitted in real time to a shared space. The environmental images are preprocessed to obtain image information, and the environmental data is preprocessed to obtain data information. The data information is converted from the radar coordinate system to the image coordinate system by the calibration matrix to obtain the point cloud projection result. A matrix of the same size as the image information is created. The pixel coordinates of the point cloud projection result are sequentially filled into the corresponding positions of the matrix of the same size. The missing values ​​are filled in by the bilateral filtering method to obtain the depth information matrix. The depth information matrix and the image information matrix are merged into perceptual information.

4. The method according to claim 1, characterized in that, The step of using a first artificial intelligence model in the backend to determine the current operational complexity and then calling matching computing resources based on that complexity specifically includes: Extract the scene change rate of continuous frames captured by high-definition cameras and the data throughput of sensor processing as judgment index features. Use a lightweight convolutional model as the first artificial intelligence model, input the judgment index features into the first artificial intelligence model, and output the current complex running situation. Computing resources are obtained from dynamically allocable units based on the current operational complexity.

5. The method according to claim 1, characterized in that, The step of using the computing resources to execute a second artificial intelligence model to predict faults based on the perceived information and obtain early warning results specifically includes: The Long Short-Term Memory Network is used as a second artificial intelligence model to output early warning results; The safety hazard level is assessed based on the warning results, and an alarm is triggered to notify the driver based on the safety hazard level.

6. The method according to claim 1, characterized in that, The specific methods for setting up human-computer interaction in the vehicle interface include: The system includes human-computer interaction methods such as touch screen, voice recognition, and gesture control. The system corrects driver deviations in the human-machine interaction method in real time based on a preset operation rule library.

7. The method according to claim 1, characterized in that, The method of using online learning algorithms to adjust the display order of each unit in the vehicle interface in real time specifically includes: The contextual slot machine algorithm is used to adjust the display order in real time. The driver's operating habits and the driving environment scene extracted from the environmental image are used as the state, the sorting of the display order is used as the action, and the score of the operation completion time is used as the reward.

8. A vehicle-mounted human-machine interaction device, characterized in that, include: The initial acquisition module is used to capture environmental images by high-definition cameras installed at the front and rear of the train, and to collect environmental data by integrating sensors. The front-end fusion module is used to align the environmental image and the environmental data in the shared space of the front end to generate perceptual information; The backend execution module is used to use a first artificial intelligence model in the backend to determine the current operational complexity, call matching computing resources according to the current operational complexity, and use the computing resources to execute a second artificial intelligence model to perform fault prediction based on the perception information, thereby obtaining an early warning result. The adaptive adjustment module is used to set the human-computer interaction mode in the vehicle interface, and to deploy the detection unit that displays the warning results, the blind spot detection unit that displays the environmental image, and the various units that control the train driving. The module uses an online learning algorithm to adjust the display order of each unit in the vehicle interface in real time.

9. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the vehicle-mounted human-machine interaction method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores an information transmission implementation program, which, when executed by a processor, implements the steps of the vehicle-mounted human-machine interaction method as described in any one of claims 1 to 7.

Images