Control method of smart device, smart device, and storage medium

Abstract

The application relates to the technical field of automatic driving, and particularly provides a control method of an intelligent device, the intelligent device and a storage medium, aiming to solve the problem of how to improve the automatic driving performance. The method comprises the following steps: inputting a data frame collected by a sensor of the intelligent device into a driving model for processing to obtain a driving decision and a planning track; controlling the intelligent device according to the driving decision and the planning track; observing a model layer in the driving model to perceive the data frame and obtain first environment features; a time sequence model layer is used for obtaining first environment state information according to the first environment features at multiple time points; a planning model layer is used for obtaining the driving decision and the planning track according to the first environment state information; and a function post-processing layer is used for post-processing the driving decision and the planning track according to a first constraint condition, which is used for constraining the driving decision and the planning track to comply with a driving rule of the intelligent device. Through the above method, the automatic driving performance of the intelligent device can be improved.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, specifically to a control method for an intelligent device, an intelligent device, and a storage medium. Background Technology

[0002] Traditional autonomous driving solutions mainly include functional modules such as perception, mapping, prediction, planning, and control. Among these modules, perception is primarily achieved using methods based on neural network models, while the other modules employ rule-based methods. For example, Figure 1 The Apollo system shown uses this traditional approach. While this autonomous driving solution boasts advantages such as mature technology, rapid deployment, and quick problem-solving, it also suffers from the following drawbacks: most functional modules rely on rule-based methods, preventing them from achieving their functions in a more human-like manner. This results in a relatively low degree of human-likeness in the overall autonomous driving functionality, failing to realistically reflect the state and decisions of human drivers. Furthermore, when using rule-based methods to implement module functions, computer program code is developed or designed based on pre-defined rules. Executing this code enables the module's functionality. Since different driving scenarios (such as highways or urban areas) require different rule settings, as driving scenarios expand, rules need to be continuously added or adjusted, leading to increasingly lengthy and difficult-to-maintain computer program code for the functional modules.

[0003] The paper "Planning-oriented Autonomous Driving" proposes another autonomous driving solution, such as... Figure 2 As shown, this autonomous driving solution employs a neural network model-based approach to implement the functions of perception, prediction, and planning. Each functional module is implemented by a separate Transformer model. Therefore, this autonomous driving solution can be understood as a modular end-to-end model composed of multiple Transformer models. This paper verifies the principle of this autonomous driving solution, demonstrating its advantages of ease of iteration, high functional ceiling, and improved anthropomorphism of autonomous driving functions. However, because this autonomous driving solution is an end-to-end model, and such end-to-end models are inherently uninterpretable, the autonomous driving decisions derived from them are also uninterpretable. Furthermore, this end-to-end model has limited representational capabilities for complex traffic environments, and cannot effectively guarantee safe vehicle operation under traffic regulations in such conditions.

[0004] Accordingly, a new technical solution is needed in this field to solve the above problems. Summary of the Invention

[0005] In order to overcome the above-mentioned deficiencies, this application is made to solve or at least partially solve the following technical problem: how to improve the anthropomorphism of autonomous driving control and effectively ensure that the vehicle can drive safely under the constraints of traffic regulations, that is, to improve the performance of autonomous driving.

[0006] In a first aspect, a method for controlling a smart device is provided, the method comprising:

[0007] Acquire data frames collected by sensors on a smart device, wherein the data frames are obtained by the sensors collecting environmental data of the environment in which the smart device is located;

[0008] The data frame is input into the intelligent driving model of the intelligent device for processing to obtain the driving decision and planning trajectory of the intelligent device;

[0009] Based on the driving decisions and the planned trajectory, the intelligent device is controlled for driving.

[0010] The intelligent driving model includes an observation model layer, a time series model layer, a planning model layer, and a functional post-processing layer.

[0011] The observation model layer includes an observation model configured to sense the data frame and obtain the first environmental features of the data frame at the corresponding time.

[0012] The time series model layer includes a time series model, which is configured to obtain first environmental state information of the environment based on first environmental features corresponding to multiple consecutive data frames at different times.

[0013] The planning model layer includes a planning model, which is configured to obtain the driving decisions and planning trajectories of the intelligent device based on the first environmental state information.

[0014] The post-processing layer is configured to post-process the driving decision and the planned trajectory to obtain a driving decision and a planned trajectory that satisfy a first constraint condition, wherein the first constraint condition is used to constrain the driving decision and the planned trajectory to conform to the driving rules of the intelligent device.

[0015] In one technical solution of the control method for the above-mentioned intelligent device, the driving rules include the traffic rules and / or safety rules of the environment;

[0016] The safety rules are the driving rules for intelligent devices in risky scenarios, and the risky scenarios are those that pose driving risks to the intelligent devices.

[0017] In one technical solution of the control method for the above-mentioned intelligent device, the first environmental feature includes obstacle features and road element features, wherein the obstacle features are the features of obstacles in the environment, and the road element features are the features of road elements in the environment;

[0018] The first environmental state information includes obstacle state information and road element state information.

[0019] In one technical solution of the above-mentioned control method for intelligent devices, the road elements include road signs, road structure, and parking areas for intelligent devices.

[0020] In one technical solution of the control method for the above-mentioned intelligent device, the timing model includes an obstacle timing model and a road timing model;

[0021] The obstacle temporal model is configured to obtain the obstacle state information based on the obstacle features at each time point in the corresponding time points of the multiple consecutive data frames;

[0022] The road time series model is configured to obtain the road element state information based on the road element features at each time point in the corresponding time points of the multiple consecutive data frames.

[0023] In one technical solution of the control method for the aforementioned intelligent device, the functional post-processing layer is configured to post-process the driving decision and planned trajectory in the following manner:

[0024] The first constraint condition includes sub-constraint conditions applied to each of the intelligent driving functions, wherein at least one intelligent driving function is activated by the intelligent device.

[0025] The driving decisions and planning trajectories used by each of the aforementioned intelligent driving functions are obtained respectively;

[0026] Based on the sub-constraints corresponding to each of the intelligent driving functions, post-processing is performed on the driving decisions and planning trajectories used by each of the intelligent driving functions.

[0027] In one technical solution of the control method for the aforementioned intelligent device, the intelligent device is a vehicle, and the intelligent driving function includes driving function, active safety function and / or parking function.

[0028] In one technical solution of the control method for the aforementioned intelligent device, the intelligent device includes a display device, and the post-processing layer is configured to:

[0029] Collect first target information for display on the display device, wherein the first target information is information that dynamically changes during the operation of the smart device;

[0030] The first target information is subjected to stability processing to obtain the second target information, and the change frequency of the second target information is less than the acquisition frequency of the first target information.

[0031] Control the display device to display the second target information.

[0032] In one technical solution of the control method for the aforementioned intelligent device, the intelligent driving model includes an observation model post-processing layer, which is disposed between the observation model layer and the time series model layer, and is configured to:

[0033] The first environmental feature output by the observation model layer is post-processed to obtain a second environmental feature that satisfies the second constraint condition, and the second environmental feature is input into the time series model layer;

[0034] The second constraint condition is used to constrain the second environmental feature to be valid input data of the time series model layer.

[0035] In one technical solution of the control method for the above-mentioned intelligent device, the second constraint includes: the confidence level of the second environmental feature is greater than a preset confidence level threshold, and / or the representation of the second environmental feature is a preset representation.

[0036] In one technical solution of the control method for the aforementioned intelligent device, the intelligent driving model includes a time-series model post-processing layer, which is disposed between the time-series model layer and the planning model layer, and is configured to:

[0037] The first environmental state information output by the time series model layer is post-processed to obtain the second environmental state information that satisfies the third constraint condition, and the second environmental state information is input into the planning model layer.

[0038] The third constraint condition is used to constrain the second environmental state information to be valid input data of the planning model layer.

[0039] In one technical solution of the control method for the aforementioned intelligent device, the third constraint condition includes: the information value of the second environmental state information is within a preset valid numerical range.

[0040] In one technical solution of the control method for the above-mentioned intelligent device, the sensor includes radar and / or visual sensors, and the observation model includes radar perception model and / or visual perception model;

[0041] The radar perception model is configured to perceive the data frames acquired by the radar and obtain the third environmental features at the corresponding time of the data frame.

[0042] The visual perception model is configured to perceive the data frames acquired by the visual sensor and obtain the fourth environmental feature at the corresponding time of the data frame.

[0043] In a second aspect, a smart device is provided, the smart device including at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program, which, when executed by the at least one processor, implements the method described in any of the technical solutions provided in the first aspect.

[0044] In a third aspect, a computer-readable storage medium is provided, wherein a plurality of program codes are stored therein, the program codes being adapted to be loaded and executed by a processor to perform the method described in any of the technical solutions provided in the first aspect above.

[0045] Solution 1. A control method for an intelligent device, characterized in that the method includes:

[0046] Acquire data frames collected by sensors on a smart device, wherein the data frames are obtained by the sensors collecting environmental data of the environment in which the smart device is located;

[0047] The data frame is input into the intelligent driving model of the intelligent device for processing to obtain the driving decision and planning trajectory of the intelligent device;

[0048] Based on the driving decisions and the planned trajectory, the intelligent device is controlled for driving.

[0049] The intelligent driving model includes an observation model layer, a time series model layer, a planning model layer, and a functional post-processing layer.

[0050] The observation model layer includes an observation model configured to sense the data frame and obtain the first environmental features of the data frame at the corresponding time.

[0051] The time series model layer includes a time series model, which is configured to obtain first environmental state information of the environment based on first environmental features corresponding to multiple consecutive data frames at different times.

[0052] The planning model layer includes a planning model, which is configured to obtain the driving decisions and planning trajectories of the intelligent device based on the first environmental state information.

[0053] The post-processing layer is configured to post-process the driving decision and the planned trajectory to obtain a driving decision and a planned trajectory that satisfy a first constraint condition, wherein the first constraint condition is used to constrain the driving decision and the planned trajectory to conform to the driving rules of the intelligent device.

[0054] Option 2. The method according to Option 1, characterized in that,

[0055] The driving rules include the traffic rules and / or safety rules of the environment;

[0056] The safety rules are driving rules for smart devices when they are in risky scenarios, and the risky scenarios are scenarios that pose driving risks to the smart devices.

[0057] Option 3. The method according to Option 1, characterized in that,

[0058] The first environmental feature includes obstacle features and road element features, wherein the obstacle features are the features of obstacles in the environment, and the road element features are the features of road elements in the environment;

[0059] The first environmental state information includes obstacle state information and road element state information.

[0060] Option 4. The method described in Option 3, wherein the road elements include road signs, road structure, and smart device parking areas.

[0061] Option 5. The method according to Option 3, wherein the time series model includes an obstacle time series model and a road time series model;

[0062] The obstacle temporal model is configured to obtain the obstacle state information based on the obstacle features at each time point in the corresponding time points of the multiple consecutive data frames;

[0063] The road time series model is configured to obtain the road element state information based on the road element features at each time point in the corresponding time points of the multiple consecutive data frames.

[0064] Solution 6. The method according to Solution 1, characterized in that the functional post-processing layer is configured to post-process the driving decision and planned trajectory in the following manner:

[0065] The first constraint condition includes sub-constraint conditions applied to each of the intelligent driving functions, wherein at least one intelligent driving function is activated by the intelligent device.

[0066] The driving decisions and planning trajectories used by each of the aforementioned intelligent driving functions are obtained respectively;

[0067] Based on the sub-constraints corresponding to each of the intelligent driving functions, post-processing is performed on the driving decisions and planning trajectories used by each of the intelligent driving functions.

[0068] Option 7. The method according to Option 6, wherein the intelligent device is a vehicle, and the intelligent driving function includes driving function, active safety function and / or parking function.

[0069] Solution 8. The method according to Solution 1, characterized in that the intelligent device includes a display device, and the functional post-processing layer is configured to:

[0070] Collect first target information for display on the display device, wherein the first target information is information that dynamically changes during the operation of the smart device;

[0071] The first target information is subjected to stability processing to obtain the second target information, and the change frequency of the second target information is less than the acquisition frequency of the first target information.

[0072] Control the display device to display the second target information.

[0073] Solution 9. The method according to any one of Solutions 1 to 8, characterized in that the intelligent driving model includes an observation model post-processing layer, the observation model post-processing layer is disposed between the observation model layer and the time series model layer, and the observation model post-processing layer is configured to:

[0074] The first environmental feature output by the observation model layer is post-processed to obtain a second environmental feature that satisfies the second constraint condition, and the second environmental feature is input into the time series model layer;

[0075] The second constraint condition is used to constrain the second environmental feature to be valid input data of the time series model layer.

[0076] Scheme 10. The method according to Scheme 9, wherein the second constraint includes: the confidence level of the second environmental feature is greater than a preset confidence level threshold, and / or the representation of the second environmental feature is a preset representation.

[0077] Solution 11. The method according to any one of Solutions 1 to 8, characterized in that the intelligent driving model includes a time-series model post-processing layer, the time-series model post-processing layer is disposed between the time-series model layer and the planning model layer, and the time-series model post-processing layer is configured to:

[0078] The first environmental state information output by the time series model layer is post-processed to obtain the second environmental state information that satisfies the third constraint condition, and the second environmental state information is input into the planning model layer.

[0079] The third constraint condition is used to constrain the second environmental state information to be valid input data of the planning model layer.

[0080] Scheme 12. The method according to Scheme 11, wherein the third constraint condition includes: the information value of the second environmental state information is within a preset valid numerical range.

[0081] Scheme 13. The method according to Scheme 1, wherein the sensor includes radar and / or a visual sensor, and the observation model includes a radar perception model and / or a visual perception model;

[0082] The radar perception model is configured to perceive the data frames acquired by the radar and obtain the third environmental features at the corresponding time of the data frame.

[0083] The visual perception model is configured to perceive the data frames acquired by the visual sensor and obtain the fourth environmental feature at the corresponding time of the data frame.

[0084] Option 14. A smart device, characterized in that it comprises:

[0085] At least one processor;

[0086] And, a memory communicatively connected to the at least one processor;

[0087] The memory stores a computer program, which, when executed by the at least one processor, implements the control method of the intelligent device as described in any one of schemes 1 to 13.

[0088] Scheme 15. A computer-readable storage medium storing a plurality of program codes, characterized in that the program codes are adapted to be loaded and run by a processor to perform a control method for an intelligent device as described in any one of Schemes 1 to 13.

[0089] The above-described technical solutions of this application have at least one or more of the following beneficial effects:

[0090] In one technical solution of the control method for the intelligent device provided in this application, the method can acquire data frames collected by sensors on the intelligent device, the data frames being obtained by sensors collecting environmental data of the environment in which the intelligent device is located; input the data frames into the intelligent driving model of the intelligent device for processing to obtain the driving decision and planning trajectory of the intelligent device; and perform driving control on the intelligent device based on the driving decision and planning trajectory.

[0091] The intelligent driving model comprises an observation model layer, a temporal model layer, a planning model layer, and a functional post-processing layer. The observation model layer includes an observation model configured to perceive data frames and obtain the first environmental features corresponding to the data frames at that time. The temporal model layer includes a temporal model configured to obtain the first environmental state information based on the first environmental features corresponding to multiple consecutive data frames at that time. The planning model layer includes a planning model configured to obtain the driving decisions and planned trajectories of the intelligent device based on the first environmental state information. The functional post-processing layer is configured to post-process the driving decisions and planned trajectories to obtain driving decisions and planned trajectories that satisfy a first constraint condition, which is used to constrain the driving decisions and planned trajectories to conform to the driving rules of the intelligent device.

[0092] In the above implementation scheme, by combining the observation model layer, the time series model layer, and the planning model layer, the driving decisions and planning trajectories of the intelligent device can be directly mapped from the data frames collected by the sensors. This establishes a direct mapping relationship between the input (data frames collected by the sensors) and the output (driving decisions and planning trajectories), achieving end-to-end acquisition of driving decisions and planning trajectories. The model structure formed by the observation model layer, the time series model layer, and the planning model layer can be understood as an end-to-end model. In this end-to-end model, the observation model layer, the time series model layer, and the planning model layer implement their respective functions based on the observation model, the time series model, and the planning model, respectively. Since the observation model, the time series model, and the planning model are all neural network models, these three model layers actually use a neural network-based method to implement their respective functions, rather than a rule-based method. Because they do not rely on rule-based methods, these three model layers can implement their functions more human-like, thereby improving the human-likeness of the entire intelligent driving model and enabling the driving decisions and planning trajectories to more realistically reflect the state and decisions of a human driver.

[0093] Because end-to-end models lack interpretability and have limited capacity to represent complex traffic environments, they cannot effectively guarantee the safe operation of intelligent devices (such as vehicles) under traffic rule constraints. However, in the aforementioned implementation scheme, a functional post-processing layer can constrain driving decisions and planned trajectories, ensuring they conform to driving rules. This allows intelligent devices to operate safely even in complex traffic environments, guided by safe driving rules. Attached Figure Description

[0094] The disclosure of this application will become more readily understood with reference to the accompanying drawings. It will be readily understood by those skilled in the art that these drawings are for illustrative purposes only and are not intended to limit the scope of protection of this application. Wherein:

[0095] Figure 1 This is a schematic diagram of existing autonomous driving solutions. Figure 1 ;

[0096] Figure 2 This is a schematic diagram of existing autonomous driving solutions. Figure 2 ;

[0097] Figure 3 This is a schematic flowchart of the main steps of a smart device control method according to an embodiment of this application;

[0098] Figure 4 This is a schematic diagram of the structure of an intelligent driving model according to an embodiment of this application. Figure 1 ;

[0099] Figure 5 This is a schematic diagram of the structure of an intelligent driving model according to an embodiment of this application. Figure 2 ;

[0100] Figure 6 This is a schematic diagram of the structure of an intelligent driving model according to an embodiment of this application. Figure 3 ;

[0101] Figure 7 This is a schematic diagram of the structure of an intelligent driving model according to an embodiment of this application. Figure 4 ;

[0102] Figure 8 This is a schematic diagram of the structure of an intelligent driving model according to an embodiment of this application. Figure 5 ;

[0103] Figure 9 This is a schematic diagram of the main structure of a smart device according to an embodiment of this application.

[0104] Figure label:

[0105] 11: Memory; 12: Processor. Detailed Implementation

[0106] Some embodiments of this application are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of this application and are not intended to limit the scope of protection of this application.

[0107] In the description of this application, "processor" can include hardware, software, or a combination of both. A processor can be a central processing unit, microprocessor, graphics processor, digital signal processor, or any other suitable processor. A processor has data and / or signal processing capabilities. A processor can be implemented in software, in hardware, or a combination of both. Computer-readable storage media includes any suitable medium capable of storing program code, such as magnetic disks, hard disks, optical disks, flash memory, read-only memory, random access memory, etc. The term "A and / or B" means all possible combinations of A and B, such as only A, only B, or A and B.

[0108] The relevant user personal information that may be involved in the various embodiments of this application is processed in strict accordance with the requirements of laws and regulations, following the principles of legality, legitimacy, and necessity, based on the reasonable purpose of the business scenario, and includes personal information that users actively provide or that is generated as a result of using the product / service, as well as personal information obtained with user authorization.

[0109] The personal information processed in this application will vary depending on the specific product / service scenario and will be based on the specific scenario in which the user uses the product / service. This may involve the user's account information, device information, driving information, vehicle information, or other related information. This application will treat the user's personal information and its processing with the utmost diligence.

[0110] This application attaches great importance to the security of users' personal information and has taken reasonable and feasible security protection measures that comply with industry standards to protect users' information and prevent unauthorized access, disclosure, use, modification, damage or loss of personal information.

[0111] The following describes an embodiment of the control method for the intelligent device provided in this application.

[0112] See appendix Figure 3 , Figure 3 This is a schematic flowchart illustrating the main steps of a smart device control method according to an embodiment of this application. Figure 3 As shown, the intelligent device control method in this application embodiment mainly includes the following steps S101 to S103.

[0113] Step S101: Obtain the data frame collected by the sensor on the smart device. The data frame is obtained by the sensor collecting environmental data of the environment where the smart device is located.

[0114] Intelligent devices can include driving equipment, intelligent vehicles, robots, and other similar devices. The environment can be, but is not limited to, scenarios such as intelligent devices driving, parking, and testing.

[0115] The smart device may also include at least one sensor for sensing information. The sensor is communicatively connected to any type of processor mentioned in this application. The sensor may include a camera, radar (such as lidar, millimeter-wave radar), etc. The sensor can be used to collect environmental data of the smart device's environment to obtain data frames. For example, a camera can collect image frames of the environment in which the smart device is located, and radar can collect point cloud frames of that environment.

[0116] Step S102: Input the data frame into the intelligent driving model of the intelligent device for processing to obtain the driving decision and planning trajectory of the intelligent device.

[0117] The intelligent driving model is a deep learning model, which can be a neural network. In other examples, the deep learning model can be a deep neural network or a convolutional neural network (CNN), or a recurrent neural network (RNN) or a transformer, etc.

[0118] Driving decisions can be understood as the driving strategies or plans of intelligent devices. For example, driving decisions can include active safety control decisions, which are used to implement active safety control of intelligent devices. Active safety control of a vehicle can include anti-lock braking (ABS) and automatic emergency braking (AEB). Taking AEB as an example, the active safety control decision can be to activate or deactivate automatic emergency braking. If automatic emergency braking is activated, the intelligent device will brake immediately. If automatic emergency braking is not activated, the intelligent device will not brake immediately.

[0119] The planned trajectory can be understood as a pre-planned driving trajectory for a smart device. When controlling the smart device, it can be controlled to drive or move according to this trajectory.

[0120] It should be noted that when obtaining driving decisions and planned trajectories through the intelligent driving model, the state information of the intelligent device can be acquired. This state information, along with the aforementioned data frames, is input into the intelligent driving model for processing, resulting in the driving decisions and planned trajectories. The aforementioned data frames reflect the environmental data of the environment in which the intelligent device is located, while the state information reflects the state of the intelligent device during its operation. The intelligent driving model processes both the environmental data and the intelligent device's own state to obtain the intelligent device's driving decisions and planned trajectories. The state information of the intelligent device may include its location information, speed, etc. For example, the vehicle's state information may include the vehicle's location information, steering wheel angle, acceleration, brake pedal information, etc.

[0121] Step S103: Based on the driving decision and planned trajectory, control the intelligent device for driving. Specifically, control the intelligent device to execute driving decisions and drive according to the planned trajectory. Taking active safety control decision as an example, if the decision is to activate automatic emergency braking, then control the intelligent device to activate automatic emergency braking, and the intelligent device will immediately perform emergency braking.

[0122] The following is in conjunction with the appendix Figure 4 The intelligent driving model in this embodiment will be described.

[0123] See appendix Figure 4 , Figure 4 The main structure of the intelligent driving model is illustrated exemplarily. For example... Figure 4 As shown, the intelligent driving model includes an observation model layer, a time series model layer, a planning model layer, and a functional post-processing layer. The following is an explanation of these three model layers and the functional post-processing layer.

[0124] 1. Explain the observation model layer.

[0125] The observation model layer can include an observation model, which can be configured to sense data frames and obtain the first environmental features at the corresponding time of the data frame.

[0126] The first environmental feature can be understood as the characteristic information of the environment in which the smart device is located. This characteristic information can reflect what targets are in the environment, and the type, location, size and other attribute information of these environmental targets. Environmental targets can include obstacles, road elements, etc.

[0127] Obstacles can include dynamic obstacles (such as vehicles and pedestrians) and static obstacles (such as traffic cones). Road elements can include road signs, road structures, and smart device parking areas. Road signs can include traffic lights and traffic signs, while road structures can include lane lines, parking lines, and road boundaries. Taking vehicles as an example, the smart device parking area is a parking space.

[0128] The observation model is a deep learning model, which can be a neural network. In some implementations, the deep learning model can be a deep neural network or a convolutional neural network (CNN), or a recurrent neural network (RNN) or a transformer, etc.

[0129] The observation model can directly map or obtain the first environmental feature from the data frame collected by the sensor, realizing the end-to-end acquisition of the first environmental feature. That is, a direct mapping relationship is established between the input (data frame collected by the sensor) and the output (first environmental feature). The observation model can be understood as an end-to-end model.

[0130] 2. Explanation of the time series model layer.

[0131] The time series model layer can include a time series model, which can be configured to obtain the first environmental state information of the environment based on the first environmental features corresponding to multiple consecutive data frames at different times.

[0132] Specifically, a data frame sequence can be obtained, comprising frame information of T data frames, where T > 1. The frame information includes the first environmental features corresponding to the time of each data frame. The frame information of the first to the Tth data frames in the data frame sequence is arranged sequentially according to the time of each data frame. The time series model can process this data frame sequence to obtain the first environmental state information.

[0133] As described above, the first environmental feature reflects the types, locations, sizes, and other attributes of the targets within the environment. The processing of the first environmental feature by the time-series model across multiple consecutive data frames can be understood as tracking and fusing the environmental targets at each of the corresponding times in the multiple data frames to obtain the state information of the current environmental target at each time point within those multiple data frames. The current environmental target can be understood as the environmental target in the last data frame of the multiple consecutive data frames, or the environmental target in the T-th data frame of the aforementioned data frame sequence. The first environmental state information can be understood as a sequence of state information of the current environmental target.

[0134] If the environmental target is an obstacle, then the state information of the environmental target can include the motion information of the obstacle, which can include the obstacle's speed, acceleration, direction, and other information.

[0135] If the environmental target is a road element, the state information of the road element can also be different for different types of road elements. For example, if the road element is a traffic light, the state information of the traffic light can include whether the traffic light is malfunctioning, whether it is obstructed, or whether it is flashing; if the road element is a parking space, the state information of the parking space can include the boundary of the parking space and whether the parking space is vacant; if the road element is a lane line, the state information of the lane line can include the direction and length of the lane line.

[0136] The temporal model layer is also a deep learning model, which can be a neural network. In some implementations, the deep learning model can be a deep neural network or a convolutional neural network (CNN), or a recurrent neural network (RNN) or a transformer, etc.

[0137] The time series model can directly map or obtain the first environmental state information from the first environmental features, realizing the end-to-end acquisition of the first environmental state information. That is, a direct mapping relationship is established between the input (first environmental features) and the output (first environmental state information). The time series model can be understood as an end-to-end model.

[0138] 3. Explain the planning model layer.

[0139] The planning model layer can include a planning model, which can be configured to obtain the driving decisions and planning trajectories of intelligent devices based on the first environmental state information.

[0140] The meaning of driving decisions and planning trajectories is described in the preceding step S102.

[0141] The planning model is also a deep learning model, which can be a neural network. In some implementations, the deep learning model can be a deep neural network or a convolutional neural network (CNN), or a recurrent neural network (RNN) or a transformer, etc.

[0142] The planning model can directly map or obtain driving decisions and planning trajectories from the first environmental state information, realizing end-to-end acquisition of driving decisions and planning trajectories. That is, a direct mapping relationship is established between the input (first environmental state information) and the output (driving decisions and planning trajectories). The planning model can be understood as an end-to-end model.

[0143] 4. Explain the function of the post-processing layer.

[0144] The post-processing layer can be configured to post-process driving decisions and planned trajectories to obtain driving decisions and planned trajectories that satisfy a first constraint. This first constraint is used to ensure that the driving decisions and planned trajectories conform to the driving rules of the intelligent device. The driving rules are established to ensure the safety of the intelligent device itself and other road users.

[0145] In some implementations, driving rules may include the traffic rules of the environment in which the smart device is located. Traffic rules are the rules governing road traffic, and all road users (including pedestrians, vehicles, etc.) must abide by them. The smart device in this application is also a road user, therefore, when controlling the smart device, it is necessary to ensure that the behavior of the smart device complies with the provisions of traffic rules, thereby ensuring the safety of the smart device itself and other road users.

[0146] In practical applications, even if the behavior of a smart device complies with traffic regulations, there may still be scenarios that pose driving risks to the smart device (hereinafter referred to as risk scenarios), endangering its safety. For example, other road users may not comply with traffic rules. Therefore, in some embodiments, driving rules may also include safety rules. Safety rules are the driving rules for smart devices in risk scenarios, ensuring that the smart device can drive safely even in risk scenarios. When setting safety rules, those skilled in the art can pre-obtain various risk scenarios and then set safety rules separately for different risk scenarios. This application does not specifically limit the types of risk scenarios or the content of safety rules.

[0147] The following is an exemplary illustration of the post-processing of driving decisions and planned trajectories.

[0148] For example, at an intersection, if the driving decision is to control the intelligent device to proceed straight through the intersection, but a pedestrian is detected about to cross the crosswalk in front of the intelligent device, according to the rule that pedestrians have the right of way, the intelligent device should yield and wait for the pedestrian to cross before proceeding straight through the intersection. Therefore, in order to make the driving decision comply with the traffic rules, the intelligent device will be controlled to stop and wait for the pedestrian to cross before proceeding straight through the intersection.

[0149] For example, in a section of a highway where vehicles merge, the road topology is complex, and a segment of the planned trajectory lies within the emergency lane. According to the rule that vehicles should not use the emergency lane unless in an emergency, vehicles will not travel in the emergency lane under non-emergency circumstances. If the vehicle is not currently in an emergency, the planned trajectory can be adjusted to ensure that the intelligent device does not travel in the emergency lane while following the planned trajectory, in order to comply with traffic regulations.

[0150] For example, when a smart device is driving in one lane, if the driving decision is to change lanes into the adjacent lane, but a vehicle is detected traveling at a high speed in the adjacent lane behind the device, changing lanes at this time may result in a collision with that vehicle. Therefore, to ensure the driving safety of the smart device, it can pause the lane change and wait for the collision risk to be eliminated before changing lanes.

[0151] For example, if a smart device has Lane Centering Control (LCC) enabled, the planned trajectory will always remain in the center of the lane. However, if a section of the trajectory is detected to be close to an obstacle that suddenly appears in the lane, the planned trajectory can be adjusted to ensure the smart device's safety while traveling along the planned trajectory, preventing a collision with the obstacle.

[0152] It should be noted that the planning model can be trained based on samples of the initial environmental state information and labeled information. The labeled information can include the driving decisions and planned trajectories of the intelligent device, and these decisions and trajectories comply with traffic regulations and ensure safe vehicle operation. Therefore, the driving decisions and planned trajectories obtained by the planning model are also obtained under the premise of complying with traffic regulations and ensuring safe driving.

[0153] However, as described in the aforementioned planning model, it is an end-to-end model. End-to-end models have limited capacity to represent complex traffic environments and may not be able to guarantee that intelligent devices (such as vehicles) drive safely under traffic rules. Therefore, in complex traffic environments, the driving decisions and planned trajectories obtained from the planning model may lead to violations of traffic rules or increase the driving risks of intelligent devices. This application, however, uses a functional post-processing layer to post-process the driving decisions and planned trajectories, ensuring that even in complex traffic environments, the driving decisions and planned trajectories conform to preset traffic and safety rules.

[0154] In the aforementioned intelligent driving model, the observation model, temporal model, and planning model are all end-to-end models. Therefore, the intelligent driving model can be understood as a modular end-to-end model. This model can directly map or obtain the driving decisions and planning trajectories of intelligent devices from the data frames collected by sensors. That is, a direct mapping relationship is established between the input (data frames collected by sensors) and the output (driving decisions and planning trajectories), realizing end-to-end acquisition of driving decisions and planning trajectories. The observation model, temporal model, and planning model are all neural network models. Therefore, the observation model layer, temporal model layer, and planning model layer in the intelligent driving model actually use neural network model-based methods to implement their respective functions, rather than rule-based methods. Because they do not rely on rule-based methods, these three model layers can implement their respective functions more human-like, thereby improving the human-likeness of the entire intelligent driving model and enabling driving decisions and planning trajectories to more realistically reflect the state and decisions of human drivers.

[0155] Furthermore, the post-processing layer can constrain driving decisions and planned trajectories, ensuring they conform to preset traffic and safety rules. This allows intelligent devices to operate safely even in complex traffic environments, effectively guaranteeing their safety and compliance.

[0156] Therefore, in the method described in steps S101 to S103 above, using the above-mentioned intelligent driving model to control the intelligent device can effectively ensure the reliability and safety of the control.

[0157] The control method for the intelligent device provided in this application will be further described below.

[0158] 1. Explain the observation model layer.

[0159] In some embodiments of this application, the environmental targets to be perceived by the observation model mainly include obstacles and road elements, the meanings of which are described in the foregoing embodiments. Based on this, the first environmental features obtained by the observation model include obstacle features and road element features, where obstacle features are the characteristics of obstacles in the environment, and road element features are the characteristics of road elements in the environment. Further, the first environmental state information obtained by the temporal model layer based on the first environmental features may include obstacle state information and road element state information.

[0160] In some implementations, the sensors installed on the smart device may include radar and / or vision sensors, where the radar may be lidar and the vision sensor may be a camera.

[0161] The observation model may include a radar perception model and / or a visual perception model.

[0162] The radar perception model can be configured to sense data frames acquired by the radar and obtain a third environmental feature at the corresponding time of the data frame. The third environmental feature has a similar meaning to the first environmental feature in the aforementioned embodiments. The main difference between the third environmental feature and the first environmental feature is that the data source of the third environmental feature is the data frame acquired by the radar, while the first environmental feature does not specify which sensor acquired the data frame.

[0163] The visual perception model can be configured to perceive data frames acquired by a visual sensor and obtain a fourth environmental feature at the corresponding moment of the data frame. The fourth environmental feature has a similar meaning to the first environmental feature in the aforementioned embodiments. The main difference between the fourth and first environmental features is that the data source of the fourth environmental feature is the data frame acquired by the visual sensor, while the first environmental feature does not specify which sensor acquired the data frame.

[0164] The first environmental feature output by the observation model layer may include the aforementioned third and / or fourth environmental features. For example, if the observation model includes both a radar perception model and a visual perception model, then the first environmental feature includes both the third and fourth environmental features; if the observation model only includes a visual perception model, then the first environmental feature only includes the fourth environmental feature.

[0165] By setting up two different perception models to perceive data frames collected by radar and vision sensors respectively, the type of driving control scheme for intelligent devices can be flexibly adjusted. If a vision-radar fusion scheme is desired, both radar perception models and visual perception models are set up simultaneously at the observation model layer; if a pure vision scheme is desired, only the radar perception model needs to be removed, and the visual perception model needs to be retained. No adjustments to the temporal model layer, planning model layer, or other structures are required, thus expanding the application scope of intelligent driving models.

[0166] 2. Explanation of the time series model layer.

[0167] See appendix Figure 5 In some embodiments of this application, the timing model may include an obstacle timing model and a road timing model. The obstacle timing model may be configured to obtain obstacle state information in the environment based on obstacle features at each time point in a series of consecutive data frames. The road timing model may be configured to obtain road element state information in the environment based on road element features at each time point in a series of consecutive data frames.

[0168] Through the above embodiments, obstacle features and road element state information can be obtained separately using two time-series models, avoiding interference and thus improving the accuracy of state information.

[0169] The obstacle time series model and the road time series model are both deep learning models.

[0170] Obstacle time series models can directly map obstacle features or obtain obstacle state information, achieving end-to-end acquisition of obstacle state information. An obstacle time series model can be understood as an end-to-end model. Similarly, road time series models can directly map road element features or obtain road element state information, achieving end-to-end acquisition of road element state information. A road time series model can also be understood as an end-to-end model.

[0171] 3. Explain the function of the post-processing layer.

[0172] In some embodiments according to this application, the functional post-processing layer can perform post-processing on driving decisions and planned trajectories according to different first constraints for different intelligent driving functions, that is, different post-processing is performed for different intelligent driving functions.

[0173] Specifically, the post-processing layer can be configured to perform the following steps:

[0174] Step 11: Obtain at least one intelligent driving function activated by the smart device. The first constraint includes sub-constraints applied to each intelligent driving function. Specifically, different sub-constraints can be pre-set for different intelligent driving functions. When post-processing of the driving decisions and planned trajectories adopted by an intelligent driving function is required, the sub-constraints corresponding to that intelligent driving function are invoked.

[0175] In this embodiment, the intelligent device can be a vehicle, and the intelligent driving function can include driving functions and / or active safety functions and / or parking functions.

[0176] Driving functions may include Adaptive Cruise Control (ACC), Lane Centering Control (LCC), Auto Lane Change (ALC), Navigate on Autopilot (NOA), and City Navigate (NOA).

[0177] Active safety features may include Automatic Emergency Braking (AEB), Anti-lock Braking (ABS), and Emergency Lane Keeping (ELK).

[0178] Parking functions can include Automated Parking Assist (APA), Remote Parking Assist (RPA), and Automated Valet Parking (AVP).

[0179] Step 12: Obtain the driving decision-making and planning trajectories used by each intelligent driving function. The driving decision-making and planning trajectories used by intelligent driving functions can be understood as the driving decision-making and planning trajectories adopted when implementing intelligent driving functions.

[0180] For example, a driving decision for active safety features could be to activate automatic emergency braking, which would then control the intelligent device to immediately apply emergency braking.

[0181] Step 13: Based on the sub-constraints corresponding to each intelligent driving function, perform post-processing on the driving decisions and planning trajectories used by each intelligent driving function.

[0182] The method for post-processing the driving decisions and planning trajectories used by the intelligent driving function based on the sub-constraints corresponding to the intelligent driving function is the same as the relevant method in the aforementioned embodiments.

[0183] Based on the methods described in steps 11 to 13 above, different post-processing is performed for different intelligent driving functions, thereby achieving more refined post-processing for each intelligent driving function.

[0184] In some embodiments according to this application, the smart device includes a display device that can display information such as text and images. Taking a vehicle as an example, the display device may be an instrument panel, a central control screen, or a HUD (Head-Up Display) in the vehicle cabin.

[0185] In this embodiment, the post-processing layer can be configured to perform the following steps:

[0186] Step 21: Collect first target information for display on the display device. The first target information is information that changes dynamically during the operation of the smart device.

[0187] The first target information may include the motion information of the smart device, such as its speed, acceleration, and orientation. Additionally, the first target information may also include the remaining charge of the device's internal battery (such as a power battery).

[0188] Step 22: Perform stability processing on the first target information to obtain the second target information. The change frequency of the second target information is less than the acquisition frequency of the first target information.

[0189] If the first target information changes frequently during the operation of the smart device (i.e., the first target information is high-frequency fluctuating information), it is very likely that the first target information collected each time will be different. If the first target information collected each time is displayed on the display device in real time, it will produce a visual effect of constantly changing information, affecting the user's viewing experience. To address this, by performing stability processing on the first target information, its change frequency can be changed to obtain second target information with a lower change frequency (i.e., the second target information is low-frequency fluctuating information). In this way, when the second target information is displayed on the display device, the visual effect of constantly changing information will not occur, thus improving the user's viewing experience.

[0190] In some implementations, stability processing can be performed in the following ways:

[0191] If the actual deviation between the first target information collected at time t and the first target information collected at time t-1 is less than a preset deviation threshold, then the information value of the second target information at time t is the same as that at time t-1. If the actual deviation is greater than or equal to the preset deviation threshold, then the information value of the second target information at time t is the information value of the first target information collected at time t. When setting the value of the deviation threshold, those skilled in the art can obtain the maximum deviation of the first target information between two adjacent times when the intelligent device is in a stable driving state, and set the value of the deviation threshold based on this maximum deviation.

[0192] For example, the first target information is the driving speed of the smart device. The first target information collected at times t1, t2, t3, t4, and t5 are 60.0 km / h, 60.1 km / h, 60.2 km / h, 60.3 km / h, and 61.0 km / h, respectively. If time t1 is the initial time, then the second target information at time t1 is 60.0 km / h. Furthermore, if the preset deviation threshold is 1 km / h, then the second target information at times t2, t3, and t4 is 60.0 km / h, and the second target information at time t5 is 61.0 km / h.

[0193] In some implementations, stability processing can be performed in the following ways:

[0194] The information value of the first target information is rounded, and the resulting information value is used as the information value of the second target information.

[0195] Taking the driving speed above as an example, if the last digit in the information value is rounded, then the first target information of 60.0km / h, 60.1km / h, 60.2km / h, 60.3km / h, and 61.0km / h collected at times t1, t2, t3, t4, and t5 respectively can be rounded to obtain 60.0km / h, 60.0km / h, 60.0km / h, 60.0km / h, and 61.0km / h.

[0196] Step 23: Control the display device to display the second target information.

[0197] Based on the methods described in steps 21 to 23 above, the display effect when displaying high-frequency fluctuation information can be improved, thereby enhancing the user's viewing experience.

[0198] The control method for the intelligent device provided in this application will be further described below.

[0199] See appendix Figure 6In some embodiments of this application, the intelligent driving model may include an observation model post-processing layer, which is disposed between the observation model layer and the time series model layer. That is, the input data of the observation model post-processing layer is the output data of the observation model layer, and the output data of the observation model post-processing layer is used as the input data of the time series model layer.

[0200] The observation model post-processing layer can be configured as follows:

[0201] The first environmental feature output from the observation model layer is post-processed to obtain a second environmental feature that satisfies the second constraint condition. This second environmental feature is then input into the time series model layer. The second constraint condition is used to ensure that the second environmental feature is valid input data for the time series model layer.

[0202] Effective input data can be understood as data that the time series model layer can process normally. Effective input data will help improve the accuracy of the output data of the time series model layer, and further guide the planning model layer to obtain safe and reliable driving decisions and planning trajectories.

[0203] The second constraint is pre-defined and is invoked directly when post-processing of the first environmental feature is required. The second constraint may include: the confidence level of the second environmental feature is greater than a preset confidence threshold, and / or the representation of the second environmental feature is a preset representation.

[0204] If the confidence level of the second environmental feature is greater than a preset confidence threshold, it indicates that the confidence level of the second environmental feature is relatively high, i.e., the degree of reliability is relatively high. Based on this constraint, environmental features with a confidence level greater than the confidence threshold from the first environmental features can be selected as the second environmental features, while environmental features with a confidence level less than or equal to the confidence threshold from the first environmental features can be removed. Environmental features with a confidence level less than or equal to the confidence threshold can be understood as noise; therefore, this constraint can be understood as removing noise from the first environmental features. When setting the confidence threshold, those skilled in the art can input first environmental features with different confidence levels into the time series model layer, evaluate which first environmental features are reliable based on the accuracy of the output data of the time series model layer, then obtain the minimum confidence level of these reliable environmental features, and set the confidence threshold based on this minimum confidence level.

[0205] The preset representation format is the representation format of input data that the temporal model layer can accept. If the representation format of the second environmental feature is the preset representation format, it indicates that the temporal model layer can recognize and process the second environmental feature. This constraint can be understood as converting the representation format of the first environmental feature into a representation format that the temporal model layer can accept. For example, if the first environmental feature is a lane line feature, which is represented by the combination of the poses of each discrete point on the lane line, and the representation format of the lane line feature that the temporal model layer can accept is a line segment, then a line segment is formed based on the poses of each discrete point in the first environmental feature, and this line segment is used as the second environmental feature.

[0206] The above embodiments can improve the signal-to-noise ratio of the input data of the time series model layer, which helps to improve the accuracy of the output data of the time series model layer. At the same time, it can also ensure that the representation of environmental features meets the requirements of the time series model layer, so that environmental features can be accurately identified and processed by the time series model layer, further improving the accuracy of the output data of the time series model layer.

[0207] Please refer to the appendix for further details. Figure 6 In some embodiments of this application, the intelligent driving model may include a time series model post-processing layer, which is disposed between the time series model layer and the planning model layer, wherein the input data of the time series model post-processing layer is the output data of the time series model layer, and the output data of the time series model post-processing layer is used as the input data of the planning model layer.

[0208] The time series model post-processing layer can be configured as follows:

[0209] The first environmental state information output from the time-series model layer is post-processed to obtain second environmental state information that satisfies the third constraint condition. This second environmental state information is then input into the planning model layer. The third constraint condition is used to ensure that the second environmental state information is valid input data for the planning model layer.

[0210] Effective input data can be understood as data that the planning model layer can process normally. Effective input data will help improve the planning model layer to obtain safe and reliable driving decisions and planning trajectories.

[0211] The third constraint is pre-defined and is invoked directly when post-processing of the first environmental state information is required. The third constraint may include: the information value of the second environmental state information is within a preset valid numerical range.

[0212] If the information value of the second environmental state information is within a preset valid value range, it indicates that the information value of the second environmental state information is normal and not abnormal. Based on this constraint, environmental state information with normal information values ​​in the first environmental state information can be filtered out as the second environmental state information, while environmental state information with abnormal information values ​​can be removed. Information values ​​outside the preset valid value range can be understood as noise; therefore, this constraint can be understood as removing noise from the first environmental state information. When setting the valid value range, those skilled in the art can obtain the normal information value range of the environmental state information and set the valid value range based on this normal information value range. Taking obstacle state information in the first environmental state information as an example, if the obstacle is a vehicle, and the obstacle state information is the vehicle's driving speed, and the normal range of driving speed is 0-180 km / h, then the valid value range of driving speed can be 0-180 km / h. For example, consider a traffic cone on a lane. The obstacle's state information includes the cone's spatial position at each time point across multiple consecutive data frames. Since the cone is a static obstacle, its spatial position does not change over time. The cone's spatial position should be within the lane's spatial range; therefore, the valid numerical range of the cone's spatial position is the spatial range of its lane. If the cone's spatial position at a certain time is outside the lane's spatial range, it indicates an anomaly and needs to be removed.

[0213] The above embodiments can improve the signal-to-noise ratio of the input data of the planning model layer, which helps to improve the accuracy of the output data of the planning model layer, and obtain safe and reliable driving decisions and planning trajectories.

[0214] The following is in conjunction with the appendix Figure 7 and attached Figure 8 This application will describe the intelligent driving model.

[0215] First, please refer to the appendix. Figure 7 In some embodiments of this application, the intelligent driving model may include a signal input layer, an observation model layer, an observation model post-processing layer, a time series model layer, a time series model post-processing layer, a planning model layer, and a functional post-processing layer. In this embodiment, the intelligent device is a vehicle, which is equipped with millimeter-wave radar, lidar, and a camera.

[0216] The signal input layer is configured to acquire millimeter-wave radar signals, lidar signals, camera signals, and positioning signals. The millimeter-wave radar signals, lidar signals, and camera signals are data frames of the vehicle's environment collected by the millimeter-wave radar, lidar, and camera, respectively. The positioning signal is a positioning signal obtained using a map.

[0217] The observation model layer includes a lidar perception model and a visual perception model.

[0218] The LiDAR perception model is configured to sense the data frames acquired by the LiDAR and obtain the third environmental features at the corresponding time. These third environmental features include obstacle features, which include dynamic obstacle features, static obstacle features, general obstacle features, drivable areas, and occupied grids. An occupied grid can be understood as the area occupied by an obstacle. Dynamic and static obstacles are categories recorded in a preset whitelist, while general obstacles are obstacle categories not recorded in the preset whitelist; that is, obstacles not recorded in the preset whitelist are considered general obstacles.

[0219] The visual perception model is configured to perceive data frames captured by the camera, obtaining the fourth environmental feature at the corresponding moment of the data frame. The fourth environmental feature includes obstacle features and road element features. Obstacle features include dynamic obstacle features, static obstacle features, general obstacle features, drivable areas, and occupied grids; the meanings of obstacle features are the same as those in the third environmental feature. Road element features include traffic light features, traffic sign features, road structure information, and parking space information. Furthermore, during perception, the visual perception model uses positioning signals to determine the positions of various environmental targets (including obstacles and road elements), thereby completing the perception using the positions of these targets and obtaining the fourth environmental feature.

[0220] The observation model post-processing layer is configured to post-process the first environmental feature output by the observation model layer to obtain a second environmental feature that satisfies the second constraint. The first environmental feature includes the third and fourth environmental features mentioned above.

[0221] The temporal model layer includes an obstacle temporal model and a road temporal model. The obstacle temporal model is configured to obtain obstacle state information based on second environmental features, and the road temporal model is configured to obtain road element state information based on second environmental features. The obstacle state information and the road element state information together constitute the first environmental state information.

[0222] The temporal model post-processing layer is configured to post-process the first environmental state information to obtain second environmental state information that satisfies the third constraint. Specifically, obstacle state information and road element state information are post-processed separately. Figure 7 In the context of obstacle post-processing, obstacle post-processing refers to post-processing obstacle state information, while road post-processing refers to post-processing road element state information.

[0223] The planning model layer includes a planning model, which is configured to acquire the driving decisions and planning trajectories of intelligent devices based on the second environmental state information.

[0224] The functional post-processing layer is configured to post-process driving decisions and planned trajectories to obtain driving decisions and planned trajectories that satisfy the first constraint condition. Specifically, post-processing is performed on the driving decisions and planned trajectories corresponding to active safety functions, driving functions, and parking functions, respectively. In addition, the functional post-processing layer can also perform visualization function post-processing, and the method for visualization function post-processing is the same as that described in steps 21 to 23 of the aforementioned embodiments.

[0225] See appendix Figure 8 , Figure 8 and Figure 7 The main difference is that, Figure 8 The observation model layer only contains a visual perception model and does not include a lidar perception model. In other words, Figure 7 This is an intelligent driving model that uses a fusion solution of vision and LiDAR. Figure 8 This is an intelligent driving model that uses a purely vision-based approach.

[0226] It should be noted that although the steps in the above embodiments are described in a specific order, those skilled in the art will understand that in order to achieve the effect of this application, different steps do not necessarily have to be executed in such an order. They can be executed simultaneously (in parallel) or in other orders. These adjusted solutions are equivalent to the technical solutions described in this application and therefore will also fall within the protection scope of this application.

[0227] Those skilled in the art will understand that all or part of the processes in the method of the above-described embodiment can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable storage medium can include any entity or device capable of carrying the computer program code, a medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory, a random access memory, an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0228] Another aspect of this application provides a computer-readable storage medium.

[0229] In one embodiment of a computer-readable storage medium according to this application, the computer-readable storage medium can be configured to store a program that performs the intelligent device control method of the above-described method embodiments. This program can be loaded and run by a processor to implement the above-described intelligent device control method. For ease of explanation, only the parts related to the embodiments of this application are shown; for specific technical details not disclosed, please refer to the method section of the embodiments of this application. The computer-readable storage medium can be a storage device comprising various electronic devices. Optionally, in the embodiments of this application, the computer-readable storage medium is a non-transitory computer-readable storage medium.

[0230] Another aspect of this application provides a smart device.

[0231] In one embodiment of a smart device according to this application, the smart device may include at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program, which, when executed by the at least one processor, implements the method described in any of the above embodiments. The smart device described in this application may include driving equipment, smart vehicles, robots, and other devices. See appendix. Figure 9 , Figure 9 The image exemplarily illustrates a communication connection between memory 11 and processor 12 via a bus.

[0232] In some embodiments of this application, the smart device may further include at least one sensor for sensing information. The sensor is communicatively connected to any type of processor mentioned in this application. Optionally, the smart device may further include an autonomous driving system for guiding the smart device to drive autonomously or assisting in driving. The processor communicates with the sensor and / or the autonomous driving system to perform the methods described in any of the above embodiments.

[0233] The technical solution of this application has been described above with reference to one embodiment shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of this application is obviously not limited to these specific embodiments. Without departing from the principles of this application, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of this application.

Claims

1. A control method for an intelligent device, characterized in that, The method includes: Acquire data frames collected by sensors on a smart device, wherein the data frames are obtained by the sensors collecting environmental data of the environment in which the smart device is located; The data frame is input into the intelligent driving model of the intelligent device for processing to obtain the driving decision and planning trajectory of the intelligent device; Based on the driving decisions and the planned trajectory, the intelligent device is controlled for driving. The intelligent driving model includes an observation model layer, a time series model layer, a planning model layer, and a functional post-processing layer. The observation model layer includes an observation model configured to sense the data frame and obtain the first environmental features of the data frame at the corresponding time. The time series model layer includes a time series model, which is configured to obtain first environmental state information of the environment based on first environmental features corresponding to multiple consecutive data frames at different times. The planning model layer includes a planning model, which is configured to obtain the driving decisions and planning trajectories of the intelligent device based on the first environmental state information. The post-processing layer is configured to post-process the driving decision and the planned trajectory to obtain a driving decision and a planned trajectory that satisfy a first constraint condition, wherein the first constraint condition is used to constrain the driving decision and the planned trajectory to conform to the driving rules of the intelligent device.

2. The method according to claim 1, characterized in that, The driving rules include the traffic rules and / or safety rules of the environment; The safety rules are driving rules for smart devices when they are in risky scenarios, and the risky scenarios are scenarios that pose driving risks to the smart devices.

3. The method according to claim 1, characterized in that, The first environmental feature includes obstacle features and road element features, wherein the obstacle features are the features of obstacles in the environment, and the road element features are the features of road elements in the environment; The first environmental state information includes obstacle state information and road element state information.

4. The method according to claim 3, characterized in that, The road elements include road signs, road structure, and smart device parking areas.

5. The method according to claim 3, characterized in that, The timing model includes an obstacle timing model and a road timing model; The obstacle temporal model is configured to obtain the obstacle state information based on the obstacle features at each time point in the corresponding time points of the multiple consecutive data frames; The road time series model is configured to obtain the road element state information based on the road element features at each time point in the corresponding time points of the multiple consecutive data frames.

6. The method according to claim 1, characterized in that, The post-processing layer is configured to post-process the driving decisions and planned trajectories in the following manner: The first constraint condition includes sub-constraint conditions applied to each of the intelligent driving functions, wherein at least one intelligent driving function is activated by the intelligent device. The driving decisions and planning trajectories used by each of the aforementioned intelligent driving functions are obtained respectively; Based on the sub-constraints corresponding to each of the intelligent driving functions, post-processing is performed on the driving decisions and planning trajectories used by each of the intelligent driving functions.

7. The method according to claim 6, characterized in that, The intelligent device is a vehicle, and the intelligent driving function includes driving functions, active safety functions, and / or parking functions.

8. The method according to claim 1, characterized in that, The smart device includes a display device, and the post-processing layer is configured to: Collect first target information for display on the display device, wherein the first target information is information that dynamically changes during the operation of the smart device; The first target information is subjected to stability processing to obtain the second target information, and the change frequency of the second target information is less than the acquisition frequency of the first target information. Control the display device to display the second target information.

9. The method according to any one of claims 1 to 8, characterized in that, The intelligent driving model includes an observation model post-processing layer, which is disposed between the observation model layer and the time series model layer, and is configured to: The first environmental feature output by the observation model layer is post-processed to obtain a second environmental feature that satisfies the second constraint condition, and the second environmental feature is input into the time series model layer; The second constraint condition is used to constrain the second environmental feature to be valid input data of the time series model layer.

10. The method according to claim 9, characterized in that, The second constraint includes: the confidence level of the second environmental feature is greater than a preset confidence threshold, and / or the representation of the second environmental feature is a preset representation.

Images