Intelligent device control methods, intelligent devices and storage media

The intelligent driving model trained by self-supervised learning uses encoding and decoding modules to process environmental videos, which solves the problems of scarce traffic participants and insufficient map information labeling in supervised learning methods, and enables intelligent devices to drive safely and reliably in various environments.

CN123119367APending Publication Date: 2026-11-17安徽蔚来智驾科技有限公司
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202510629130.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-05-15
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

In existing technologies, when vehicles are driving intelligently in urban environments, the supervised learning methods cannot effectively label rare traffic participants and map information, resulting in insufficient reliability of intelligent driving.

Method used

The intelligent driving model employs self-supervised training. It collects environmental videos through cameras and uses encoding, deep learning, and decoding modules for self-supervised learning. The training dataset contains videos from various driving environments, and the model is optimized to improve driving reliability in different environments.

Benefits of technology

It enables safe and reliable driving control of intelligent devices in various driving environments, simulates human driving behavior, and improves the accuracy and safety of intelligent driving.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123119367A_ABST
    Figure CN123119367A_ABST
Patent Text Reader

Abstract

This application relates to the field of intelligent driving technology, specifically providing an intelligent device control method, an intelligent device, and a storage medium, aiming to solve the problem of improving the reliability of intelligent driving in different driving environments. To this end, the method provided in this application includes acquiring a first environmental video captured by a camera on the intelligent device; inputting the first environmental video into an intelligent driving model for processing to obtain a planned trajectory of the intelligent device and a second environmental video, where each frame in the second environmental video represents a predicted image of the driving environment at multiple future moments; controlling the intelligent device according to the planned trajectory; and training the intelligent driving model using a self-supervised training dataset. Through the above method, the intelligent driving model can safely and reliably control the intelligent device in various driving environments.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of intelligent driving technology, specifically to an intelligent device control method, an intelligent device, and a storage medium. Background Technology

[0002] When a vehicle is driving intelligently, it needs to accurately perceive map information in the driving environment (such as lanes, traffic signs, etc.), and also needs to correctly predict the movement of all traffic participants and plan the trajectory based on the movement of traffic participants.

[0003] Currently, vehicles may exhibit poor performance when performing intelligent driving in urban environments. Research has identified the main reasons for this issue: Intelligent driving typically utilizes models for perception and trajectory planning. However, current model training methods primarily rely on supervised learning. Supervised learning methods cannot label all map information and traffic participants in driving environment data (such as images). Some rare traffic participants and map information are easily overlooked during labeling, and the movement of traffic participants during traffic butterfly effects cannot be captured. When using supervised learning to train models, the models can only learn the labeled information from the driving environment data. Therefore, when map information and traffic participant information that the model has not learned are present in the driving environment, the reliability of intelligent driving may be affected, leading to poor performance.

[0004] Accordingly, a new technical solution is needed in this field to solve the above problems. Summary of the Invention

[0005] In order to overcome the above-mentioned deficiencies, this application is made to solve or at least partially solve the following technical problem: how to improve the reliability of intelligent driving of vehicles in different driving environments (including at least urban environments).

[0006] In a first aspect, a method for controlling a smart device is provided, the method being applied to a smart device having a camera installed, the method comprising:

[0007] The first environmental video obtained by the camera capturing the driving environment is acquired, wherein the driving environment is the environment in which the smart device is located;

[0008] The first environmental video is input into the intelligent driving model of the intelligent device for processing to obtain the planned trajectory of the intelligent device and the second environmental video. The planned trajectory includes the trajectory information of the intelligent device at multiple future moments, and each frame of the second environmental video is a predicted image of the driving environment at each of the multiple future moments.

[0009] The intelligent device is driven according to the planned trajectory;

[0010] The intelligent driving model is obtained through self-supervised training based on a training dataset, which includes environmental videos collected by the camera of the intelligent device in various driving environments.

[0011] In one technical solution of the above-mentioned intelligent device control method, the intelligent driving model includes an encoding module, a deep learning model, a driving decoding module, and an imagination decoding module;

[0012] The encoding module is configured to: encode the first environmental video to obtain encoded information;

[0013] The deep learning model is configured to decode the encoded information to obtain the environmental features of the driving environment.

[0014] The driving decoding module is configured to decode the environmental features to obtain the planned trajectory.

[0015] The imagination decoding module is configured to decode the environmental features to obtain the second environmental video.

[0016] In one technical solution of the above-mentioned intelligent device control method, the intelligent driving model is trained in the following way:

[0017] Obtain a first loss function for self-supervised learning of the planned trajectory;

[0018] Obtain a second loss function for self-supervised learning of the second environment video;

[0019] The first loss function and the second loss function are weighted and summed to form the target loss function;

[0020] The intelligent driving model is self-supervised trained based on the target loss function and the training dataset.

[0021] In one technical solution of the above-mentioned intelligent device control method, when performing self-supervised training on the intelligent driving model, the model evaluation index of the intelligent driving model includes the video duration and content diversity level of the second environmental video. The content diversity level is positively correlated with the number of traffic participants appearing in the second environmental video, and the traffic participants include at least vehicles.

[0022] Specifically, after the intelligent driving model completes self-supervised training, the duration of the second environmental video obtained by the intelligent driving model is at least a preset duration.

[0023] In one technical solution of the above-mentioned intelligent device control method, the content diversity level includes slightly, moderate, good and significant levels, which increase in order of increasing level;

[0024] And / or, the preset duration is 2 minutes.

[0025] In one technical solution of the above-mentioned intelligent device control method, after the intelligent driving model completes self-supervised training, the intelligent driving model is tested in the following way:

[0026] Obtain a test dataset, which includes environmental videos collected by the camera of the smart device under various risky environments, wherein the risky environment is a driving environment that poses a driving risk to the smart device;

[0027] The intelligent driving model is tested using the test dataset, and the dynamic interaction process between the intelligent device and the risk environment is demonstrated through the environmental video output by the intelligent driving model during the test, so as to form a closed-loop test of the intelligent driving model.

[0028] In one technical solution of the above-mentioned intelligent device control method, the trajectory information includes the position of the trajectory point and the state information of the intelligent device.

[0029] In one technical solution of the above-mentioned intelligent device control method, the method further includes: inputting the first environmental video into the intelligent driving model of the intelligent device for processing to obtain semantic information of driving decision corresponding to the planned trajectory.

[0030] In a second aspect, a smart device is provided, the smart device comprising at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program that, when executed by the at least one processor, implements the method described in any of the technical solutions provided in the first aspect.

[0031] In a third aspect, a computer-readable storage medium is provided, wherein a plurality of program codes are stored therein, the program codes being adapted to be loaded and executed by a processor to perform the method described in any of the technical solutions provided in the first aspect.

[0032] The above-described technical solutions of this application have at least one or more of the following features. Beneficial effects:

[0033] In one technical solution of the intelligent device control method provided in this application, the method is applied to an intelligent device equipped with a camera. The method acquires a first environmental video captured by the camera of the driving environment, where the driving environment is the environment in which the intelligent device is located. The first environmental video is input into the intelligent driving model of the intelligent device for processing to obtain a planned trajectory and a second environmental video. The planned trajectory includes trajectory information of the intelligent device at multiple future moments, and each frame of the second environmental video is a predicted image of the driving environment at each of the multiple future moments. The intelligent device is controlled for driving based on the planned trajectory. The intelligent driving model is obtained through self-supervised training based on a training dataset, which includes environmental videos captured by the camera of the intelligent device in various different driving environments.

[0034] Human drivers typically follow three steps when driving intelligent devices (such as vehicles): seeing, imagining, and doing. Seeing involves observing the current driving environment; imagining involves predicting the future driving environment based on the observed environment (e.g., predicting the movement of traffic participants); and doing involves deciding on driving behavior and controlling the intelligent device based on the observed and imagined future driving environments. In the above implementation scheme, acquiring the first environmental video is equivalent to simulating the seeing step, predicting the second environmental video is equivalent to simulating the imagining step, and predicting the planned trajectory is equivalent to simulating the doing step. In other words, the above implementation scheme simulates human driving behavior by having the intelligent device first observe the driving environment, then imagine what will happen in the future based on the observed environment (i.e., predict the future driving environment), and make decisions accordingly to determine the planned trajectory of the intelligent device. Driving control is then performed based on the planned trajectory. This means that the intelligent device, as a physical entity, perceives, understands, makes, and executes the driving environment, achieving embodied intelligence. The above implementation scheme is a method for controlling intelligent devices based on embodied intelligence.

[0035] The intelligent driving model is trained under self-supervised conditions based on a training dataset, enabling it to fully learn various information from the driving environment. This avoids the problem in existing technologies where supervised learning methods only learn labeled information from the driving environment data. Furthermore, the training dataset includes environmental videos captured by the smart device's camera in various driving environments (at least urban environments). Therefore, after training the intelligent driving model using this dataset, it can safely and reliably control the smart device in a variety of driving environments. Attached Figure Description

[0036] The disclosure of this application will become more readily understood with reference to the accompanying drawings. It will be readily understood by those skilled in the art that these drawings are for illustrative purposes only and are not intended to limit the scope of protection of this application. Wherein:

[0037] Figure 1 This is a schematic flowchart of the main steps of a smart device control method according to an embodiment of this application;

[0038] Figure 2 This is a schematic diagram of the main structure of an intelligent driving model according to an embodiment of this application;

[0039] Figure 3 This is a schematic flowchart of the main steps of an intelligent driving model training method according to an embodiment of this application;

[0040] Figure 4 This is a second environmental video illustration according to an embodiment of this application. Figure 1 ;

[0041] Figure 5 This is a schematic flowchart of the main steps of an intelligent driving model testing method according to an embodiment of this application;

[0042] Figure 6 This is a second environmental video illustration according to an embodiment of this application. Figure 2 ;

[0043] Figure 7 This is a schematic diagram of the main structure of a smart device according to an embodiment of this application.

[0044] Figure label:

[0045] 11: Memory; 12: Processor. Detailed Implementation

[0046] Some embodiments of this application are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of this application and are not intended to limit the scope of protection of this application.

[0047] In the description of this application, "processor" can include hardware, software, or a combination of both. A processor can be a central processing unit, microprocessor, graphics processor, digital signal processor, or any other suitable processor. A processor has data and / or signal processing capabilities. A processor can be implemented in software, in hardware, or a combination of both. Computer-readable storage media includes any suitable medium capable of storing program code, such as magnetic disks, hard disks, optical disks, flash memory, read-only memory, random access memory, etc. The term "A and / or B" means all possible combinations of A and B, such as only A, only B, or A and B.

[0048] The relevant user personal information that may be involved in the various embodiments of this application is processed in strict accordance with the requirements of laws and regulations, following the principles of legality, legitimacy, and necessity, based on the reasonable purpose of the business scenario, and includes personal information that users actively provide or that is generated as a result of using the product / service, as well as personal information obtained with user authorization.

[0049] The personal information processed in this application will vary depending on the specific product / service scenario and will be based on the specific scenario in which the user uses the product / service. This may involve the user's account information, device information, driving information, vehicle information, or other related information. This application will treat the user's personal information and its processing with the utmost diligence.

[0050] This application attaches great importance to the security of users' personal information and has taken reasonable and feasible security protection measures that comply with industry standards to protect users' information and prevent unauthorized access, disclosure, use, modification, damage or loss of personal information.

[0051] The following describes embodiments of the intelligent device control method provided in this application. This intelligent device control method is applied to intelligent devices, meaning it is executed by the intelligent device. Intelligent devices may include driving devices, intelligent vehicles, robots, etc. The intelligent device is equipped with a camera, which can capture video or images of the environment in which the intelligent device is located.

[0052] See appendix Figure 1 , Figure 1 This is a schematic flowchart illustrating the main steps of a smart device control method according to an embodiment of this application. Figure 1 As shown, the intelligent device control method in this application embodiment mainly includes the following steps S101 to S103.

[0053] Step S101: Acquire a first environmental video obtained by the camera capturing the driving environment, where the driving environment is the environment in which the smart device is located. The first environmental video consists of multiple consecutive frames.

[0054] Step S102: Input the first environmental video into the intelligent driving model of the intelligent device for processing to obtain the planned trajectory of the intelligent device and the second environmental video.

[0055] The planned trajectory can include trajectory information of the smart device at multiple points in the future. This trajectory information can include the locations of trajectory points and the state information of the smart device; that is, the planned trajectory includes the locations of trajectory points at each point in the future, as well as the state information of the smart device at each point in time. State information can include the smart device's speed, acceleration, etc. Taking a vehicle as an example, the state information can include the vehicle's speed, acceleration, steering wheel angle, etc.

[0056] Each frame in the second environment video is a predicted image of the driving environment at each of the aforementioned future moments. The predicted images are not actually captured by the camera, but are predicted by the intelligent driving model based on the first environment video. The second environment video can be understood as the future driving environment imagined by the intelligent driving model based on the observed first environment video.

[0057] It should be noted that when acquiring the planned trajectory and the second environmental video through the intelligent driving model, the status information of the intelligent device can be obtained. This status information is then input into the intelligent driving model along with the first environmental video. The first environmental video reflects the environment in which the intelligent device is located, while the status information reflects the intelligent device's state while driving in that environment. The intelligent driving model processes both the first environmental video and the intelligent device's own state to obtain the planned trajectory and the second environmental video. The status information may include the intelligent device's location information, speed, etc.

[0058] The intelligent driving model is trained under self-supervised conditions using a training dataset, which includes environmental videos captured by cameras on intelligent devices under various driving environments. The driving environment includes at least an urban environment, which can encompass the urban environments of multiple cities under various conditions, including different weather, lighting, and seasons. For example, for a single city, environmental videos can be captured by cameras on intelligent devices during daytime, nighttime, and all four seasons. In some implementations, the training dataset contains environmental videos from over 700 cities, with a total duration of 30,000 hours.

[0059] Furthermore, the environmental videos in the training dataset include footage captured by cameras in risky environments. Risky environments are driving conditions that pose risks to intelligent devices, often requiring complex driving behaviors and rapid response operations. Adding these risky environment videos to the training dataset and using them for self-supervised training of the intelligent driving model enables the model to obtain safe and reliable trajectory planning even in risky environments, ensuring safe driving for intelligent devices. Taking a vehicle as an example, risky environments can include driving scenarios such as lane changing, overtaking, and cutting in.

[0060] In addition, in this embodiment, the conventional self-supervised learning method can be used to perform self-supervised training on the intelligent driving model. The specific principles and processes of self-supervised training will not be elaborated in this embodiment.

[0061] Step S103: Control the driving of the intelligent device according to the planned trajectory.

[0062] Specifically, it controls intelligent devices to travel along a planned trajectory.

[0063] Based on the methods described in steps S101 to S103 above, an intelligent device can simulate human driving behavior. First, it observes the driving environment, then imagines what might happen in the future based on the observed environment, and makes decisions accordingly to determine the planned trajectory of the intelligent device. Driving control is then performed based on this planned trajectory. Specifically, acquiring the first environmental video is equivalent to simulating human observation of the driving environment, predicting the second environmental video is equivalent to simulating human imagination of what might happen in the future, and predicting the planned trajectory is equivalent to simulating human decision-making based on the observed and imagined future driving environments to control the intelligent device. Based on this, intelligent device control based on embodied intelligence is achieved.

[0064] In the above method, the intelligent driving model is trained under self-supervised conditions based on the training dataset. This allows it to fully learn various information from the driving environment, avoiding the problem in existing technologies where supervised learning methods only learn labeled information from the driving environment data. The training dataset includes environmental videos collected by the smart device's camera in various driving environments. After training the intelligent driving model using this dataset, it can safely and reliably control the smart device in various driving environments.

[0065] Based on the embodiments of the intelligent device control method provided in this application, safe and reliable driving control of vehicles can be achieved in application scenarios such as urban road navigation assistance, highway road navigation assistance, intelligent parking, and parking lot wayfinding.

[0066] Taking parking lot navigation as an example, when a vehicle is located within a parking area and given a destination (such as the parking lot exit, other parking areas, etc.), the intelligent device control method provided in this application can control the vehicle to autonomously find and drive to the destination. Specifically, the vehicle's camera captures a first environmental video of the parking lot where the intelligent device is located. This first environmental video records multimodal environmental information such as signs and text within the parking lot. The intelligent driving model possesses spatiotemporal reasoning capabilities. By performing temporal and spatial reasoning on the first environmental video, the intelligent driving model can read and understand this multimodal environmental information of the parking lot and derive a planned trajectory. Based on the planned trajectory, the vehicle can be safely and reliably controlled to drive to the destination. During this process, no reliance on any form of map, navigation, or route memory is required. Even if the vehicle's current location is in a different parking area, on a different floor, or in a different building than the destination, the vehicle can still be safely and reliably controlled to drive to the destination.

[0067] The embodiments of the intelligent device control method provided in this application will be described below, specifically the intelligent driving model.

[0068] See appendix Figure 2 , Figure 2 The main structure of the intelligent driving model is illustrated exemplarily. For example... Figure 2 As shown, the intelligent driving model includes an encoding module, a deep learning model, a driving decoding module, and an imagination decoding module. Specifically, the encoding module can be configured to encode a first environmental video to obtain encoded information. The deep learning model can be configured to decode the encoded information to obtain environmental features of the driving environment, which incorporate spatiotemporal knowledge from the first environmental video. The driving decoding module can be configured to decode the environmental features to obtain a planned trajectory. The imagination decoding module can be configured to decode the environmental features to obtain a second environmental video.

[0069] The driving decoding module and the imagination decoding module can be understood as two branches on the output side of the deep learning model: the driving branch and the imagination branch. These modules run in parallel within the intelligent driving model. However, since their inputs are connected to the output of the deep learning model, during self-supervised training, the model parameters are actually optimized based on the outputs of both modules. Therefore, after self-supervised training, the environmental features output by the deep learning model simultaneously address the decoding needs of both modules. This allows the driving decoding module to obtain a safe and reliable planned trajectory, while also enabling the imagination decoding module to predict an accurate secondary environmental video. It's understandable that if the environmental features address the decoding needs of both modules, then the planned trajectory obtained by the driving decoding module must also take into account the decoding needs of the imagination decoding module. Therefore, this planned trajectory is essentially derived from both the observed driving environment and the imagined future driving environment.

[0070] In some implementations, the deep learning model can be a diffusion model or a transformer.

[0071] The following describes embodiments of the intelligent device control method provided in this application, specifically the training of the intelligent driving model.

[0072] In some embodiments according to this application, it is possible to... Figure 3 The intelligent driving model is obtained by training through the following steps S201 to S203.

[0073] Step S201: Obtain the first loss function L for self-supervised learning of the planned trajectory. driveObtain the second loss function L for self-supervised learning of the second environment video. imagine Step S202: For the first loss function L drive With the second loss function L imagine Weighted summation is performed to form the target loss function L. L = λ₁L drive +λ2L imagine λ1 and λ2 represent weights. Step S203: Perform self-supervised training on the intelligent driving model based on the target loss function L and the training dataset. By adjusting the values ​​of weights λ1 and λ2, the proportions of the loss values ​​of the first and second loss functions in the loss value of the target loss function can be adjusted. Different proportions indicate different importance of the planned trajectory prediction and the second environmental video prediction in model training. Based on this, those skilled in the art can flexibly adjust the values ​​of weights λ1 and λ2 according to the actual requirements for the importance of the planned trajectory and the second environmental video. In some implementations, λ1 > λ2.

[0074] In this embodiment, the first and second loss functions can be constructed using conventional loss functions in the field of self-supervised learning, and this embodiment does not impose specific limitations on them. For example, in some implementations, the first loss function is the mean square error loss function, and the second loss function is the cross entropy loss function.

[0075] In some embodiments of this application, when performing self-supervised training on the intelligent driving model, the model evaluation metrics include the video duration and content diversity level of the second environment video. The content diversity level is positively correlated with the number of traffic participants appearing in the second environment video; that is, the more traffic participants, the higher the content diversity level. Traffic participants include at least vehicles. In some embodiments, the content diversity level includes four levels: slightly, moderate, good, and significant, with levels increasing sequentially. In some embodiments, when the intelligent device is a vehicle, the content diversity level can also be positively correlated with the number of scenes appearing in the second environment video; that is, the more scenes, the higher the content diversity level. Scenes appearing in the video can include vehicles driving straight, turning, making U-turns, changing lanes, overtaking, and cutting in.

[0076] In this embodiment, after the intelligent driving model completes self-supervised training, the second environment video obtained by the intelligent driving model has a video duration of at least a preset duration. The preset duration is a relatively large value to ensure that the second environment video is not too short. In some implementations, the preset duration is 2 minutes, meaning at least 120 seconds of second environment video is obtained. For example, Figure 4An example is shown, consisting of partial image frames from a 2-minute second environment video. Using the video duration as a model evaluation metric, and through self-supervised training, the intelligent driving model only needs to be input into a short environment video to predict a longer second environment video, thus obtaining a longer second environment video in a one-shot prompt manner.

[0077] The following describes embodiments of the intelligent device control method provided in this application, specifically the testing of the intelligent driving model.

[0078] In some embodiments according to this application, it is possible to... Figure 5 The following steps S301 to S302 are shown to test the intelligent driving model.

[0079] Step S301: Obtain the test dataset, which includes environmental videos captured by the smart device's camera under various risky conditions. The risky environment refers to the driving environment that poses a driving risk to the smart device. Taking a vehicle as an example, the risky environment can include driving scenarios such as lane changing, overtaking, and cutting in.

[0080] Step S302: Test the intelligent driving model using the test dataset, and demonstrate the dynamic interaction between the intelligent device and the risky environment through the environmental video output by the intelligent driving model during the test, so as to form a closed-loop test of the intelligent driving model.

[0081] Based on the methods described in steps S301 to S302 above, there is no need to construct a separate closed-loop test simulator; the intelligent driving model itself can achieve the function of a closed-loop test simulator. Specifically, the environmental video input to the intelligent driving model during model testing (i.e., the environmental video in the test dataset) is equivalent to the simulated scenario in the closed-loop test. The environmental video output by the intelligent driving model is equivalent to the scene observed by the intelligent device in the simulated scenario. This environmental video can demonstrate the dynamic interaction process between the intelligent device and the simulated scenario, and the planned trajectory output by the intelligent driving model is equivalent to the driving trajectory of the intelligent device in the simulated scenario.

[0082] The following example illustrates the testing effect of the intelligent driving model using a scenario where a car is parked and occupies a lane. The intelligent device is a vehicle (hereinafter referred to as the "automobile"). In this scenario, there is another vehicle (hereinafter referred to as an obstacle) parked and occupying the lane in front of the vehicle, and there is another vehicle (hereinafter referred to as the "front vehicle") in front of the vehicle. Due to the obstacle, the vehicle needs to use the oncoming lane to bypass it. There are three possibilities: (1) The front vehicle successfully uses the oncoming lane to bypass the obstacle, and the vehicle can follow closely behind the front vehicle to bypass the obstacle; (2) The front vehicle stops, and the field of vision is open, and the vehicle can use the oncoming lane to bypass the obstacle; (3) The front vehicle fails to use the oncoming lane to bypass the obstacle and stops to avoid oncoming traffic, and the vehicle can stop behind the front vehicle to avoid oncoming traffic.

[0083] Testing has confirmed that the intelligent driving model can predict the potential behaviors of other road users in a given scenario and trigger safe and reliable vehicle operation based on environmental changes caused by these behaviors. Furthermore, the intelligent driving model can easily handle risky environments, ensuring the safe and reliable operation of intelligent devices even under such conditions. For example, Figure 6 The example shows partial images from three second-environment videos (i.e., imagined scenarios 1, 2, and 3) output by the intelligent driving model. All three videos depict risky environments, but the vehicle is able to drive safely in any of the video environments without colliding with other road users.

[0084] In some embodiments of this application, two real-world accident cases are used to test the intelligent driving model. One case involves a collision between the vehicle and another vehicle caused by a human changing lanes; the other case involves a collision between the vehicle and another vehicle caused by a human cutting in front of another vehicle. The videos of the events preceding the collisions in these two cases (i.e., the first environmental videos) are input into the intelligent driving model. Based on the planned trajectory derived from the videos, the intelligent driving model can avoid collisions with other vehicles. Furthermore, the second environmental video output by the intelligent driving model demonstrates that the vehicle can drive safely and will not collide with other vehicles.

[0085] The embodiments of the intelligent device control method provided in this application will be described below. In some embodiments of this application, a first environmental video is input to the intelligent driving model of the intelligent device for processing. This not only yields a planned trajectory but also semantic information about the driving decisions corresponding to the planned trajectory. For example, the semantic information describes the content of the driving decisions, such as following closely behind the vehicle in front, using the oncoming lane to bypass obstacles, etc. After obtaining the semantic information, it can be output through text, images, or voice, helping the user understand the driving decisions of the intelligent device.

[0086] It should be noted that although the steps in the above embodiments are described in a specific order, those skilled in the art will understand that in order to achieve the effect of this application, different steps do not necessarily have to be executed in such an order. They can be executed simultaneously (in parallel) or in other orders. These adjusted solutions are equivalent to the technical solutions described in this application and therefore will also fall within the protection scope of this application.

[0087] Those skilled in the art will understand that all or part of the processes in the method of the above-described embodiment can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable storage medium can include any entity or device capable of carrying the computer program code, a medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory, a random access memory, an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0088] Another aspect of this application provides a computer-readable storage medium.

[0089] In one embodiment of a computer-readable storage medium according to this application, the computer-readable storage medium can be configured to store a program that performs the intelligent device control method of the above-described method embodiments. This program can be loaded and run by a processor to implement the above-described method. For ease of explanation, only the parts related to the embodiments of this application are shown; for specific technical details not disclosed, please refer to the method section of the embodiments of this application. The computer-readable storage medium can be a storage device comprising various electronic devices. Optionally, in the embodiments of this application, the computer-readable storage medium is a non-transitory computer-readable storage medium.

[0090] Another aspect of this application provides a smart device.

[0091] In one embodiment of a smart device according to this application, the smart device may include at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program, which, when executed by the at least one processor, implements the method described in any of the above embodiments. The smart device described in this application may include driving equipment, smart vehicles, robots, and other devices. See appendix. Figure 7 , Figure 7 The image exemplarily illustrates a communication connection between memory 11 and processor 12 via a bus.

[0092] In some embodiments of this application, the smart device may further include at least one sensor for sensing information. The sensor is communicatively connected to any type of processor mentioned in this application. Optionally, the smart device may further include an intelligent driving system for guiding the smart device in assisted driving. The processor communicates with the sensor and / or the intelligent driving system to perform the methods described in any of the above embodiments.

[0093] The technical solution of this application has been described above with reference to one embodiment shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of this application is obviously not limited to these specific embodiments. Without departing from the principles of this application, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of this application.

Claims

1. A method for controlling an intelligent device, characterized in that, The method is applied to a smart device equipped with a camera, and the method includes: The first environmental video obtained by the camera capturing the driving environment is acquired, wherein the driving environment is the environment in which the smart device is located; The first environmental video is input into the intelligent driving model of the intelligent device for processing to obtain the planned trajectory of the intelligent device and the second environmental video. The planned trajectory includes the trajectory information of the intelligent device at multiple future moments, and each frame of the second environmental video is a predicted image of the driving environment at each of the multiple future moments. The intelligent device is driven according to the planned trajectory; in, The intelligent driving model is obtained through self-supervised training based on a training dataset, which includes environmental videos collected by the camera of the intelligent device in various driving environments.

2. The method according to claim 1, characterized in that, The intelligent driving model includes an encoding module, a deep learning model, a driving decoding module, and an imagination decoding module; The encoding module is configured to: encode the first environmental video to obtain encoded information; The deep learning model is configured to decode the encoded information to obtain the environmental features of the driving environment. The driving decoding module is configured to decode the environmental features to obtain the planned trajectory. The imagination decoding module is configured to decode the environmental features to obtain the second environmental video.

3. The method according to claim 1, characterized in that, The intelligent driving model was trained in the following way: Obtain a first loss function for self-supervised learning of the planned trajectory; Obtain a second loss function for self-supervised learning of the second environment video; The first loss function and the second loss function are weighted and summed to form the target loss function; The intelligent driving model is self-supervised trained based on the target loss function and the training dataset.

4. The method according to claim 1 or 3, characterized in that, When performing self-supervised training on the intelligent driving model, the model evaluation metrics of the intelligent driving model include the video duration and content diversity level of the second environmental video. The content diversity level is positively correlated with the number of traffic participants appearing in the second environmental video, and the traffic participants include at least vehicles. Specifically, after the intelligent driving model completes self-supervised training, the duration of the second environmental video obtained by the intelligent driving model is at least a preset duration.

5. The method according to claim 4, characterized in that, The content diversity levels are categorized as slightly, moderate, good, and significant, with the levels increasing sequentially. And / or, the preset duration is 2 minutes.

6. The method according to claim 1, characterized in that, After completing self-supervised training, the intelligent driving model is tested using the following methods: Obtain a test dataset, which includes environmental videos collected by the camera of the smart device under various risky environments, wherein the risky environment is a driving environment that poses a driving risk to the smart device; The intelligent driving model is tested using the test dataset, and the dynamic interaction process between the intelligent device and the risk environment is demonstrated through the environmental video output by the intelligent driving model during the test, so as to form a closed-loop test of the intelligent driving model.

7. The method according to claim 1, characterized in that, The trajectory information includes the location of the trajectory points and the status information of the smart device.

8. The method according to claim 1, characterized in that, The method further includes: The first environmental video is input into the intelligent driving model of the intelligent device for processing to obtain the semantic information of the driving decision corresponding to the planned trajectory.

9. A smart device, characterized in that, include: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores a computer program, which, when executed by the at least one processor, implements the intelligent device control method according to any one of claims 1 to 8.

10. A computer-readable storage medium storing a plurality of program codes, characterized in that, The program code is adapted to be loaded and run by a processor to perform the intelligent device control method according to any one of claims 1 to 8.