Multi-task prediction model training method, mobile device control method and apparatus

Abstract

A multi-task prediction model training method, a mobile device control method and device are disclosed. The method comprises: obtaining a first sequence and a second sequence; based on the first sequence and the second sequence, generating obstacle prediction data corresponding to each of a plurality of prediction tasks via a multi-task prediction network; for each of the plurality of prediction tasks, determining a task loss value corresponding to the prediction task based on the obstacle prediction data corresponding to the prediction task and a corresponding loss calculation manner; determining a multi-task loss value based on the task loss value corresponding to each of the plurality of prediction tasks; training the multi-task prediction network based on the multi-task loss value; and in response to the trained multi-task prediction network meeting a preset training end condition, determining the trained multi-task prediction network as a multi-task prediction model. The disclosed embodiments can improve the reliability of autonomous driving.

Description

Technical Field

[0001] This disclosure relates to driving technology, and in particular to a multi-task prediction model training method, a control method and device for mobile devices. Background Technology

[0002] Autonomous driving technology is being used more and more widely in vehicles and other mobile devices. Generally speaking, vehicles control their driving based on data collected by sensors such as cameras and radar installed on their own, thereby achieving autonomous driving. Summary of the Invention

[0003] To address the current technical problem of low reliability in autonomous driving, this disclosure is proposed. Embodiments of this disclosure provide a multi-task prediction model training method, a mobile device control method, and an apparatus.

[0004] According to one aspect of the present disclosure, a method for training a multi-task prediction model is provided, comprising:

[0005] Obtain a first sequence and a second sequence, wherein the first sequence includes: a local map of a local area corresponding to a first mobile device at each of the multiple time points, and the second sequence includes: obstacle information around the first mobile device at each of the multiple time points;

[0006] Based on the first sequence and the second sequence, obstacle prediction data corresponding to each of the multiple prediction tasks is generated via a multi-task prediction network;

[0007] For each of the multiple prediction tasks, the task loss value corresponding to the prediction task is determined based on the obstacle prediction data corresponding to the prediction task and the corresponding loss calculation method.

[0008] Based on the task loss values ​​corresponding to each of the multiple prediction tasks, determine the multi-task loss value;

[0009] The multi-task prediction network is trained based on the multi-task loss value;

[0010] In response to the trained multi-task prediction network meeting the preset training termination condition, the trained multi-task prediction network is determined as a multi-task prediction model.

[0011] According to another aspect of the present disclosure, a method for controlling a mobile device is provided, comprising:

[0012] A third sequence and a fourth sequence are obtained, wherein the third sequence includes a local map of a local area corresponding to the second mobile device at each of the multiple time points, and the fourth sequence includes obstacle information around the second mobile device at each of the multiple time points.

[0013] Based on the third sequence and the fourth sequence, obstacle prediction data corresponding to each of the multiple prediction tasks are generated through a multi-task prediction model.

[0014] Based on the obstacle prediction data corresponding to each of the multiple prediction tasks, the second mobile device is controlled for driving.

[0015] According to another aspect of the present disclosure, a multi-task prediction model training apparatus is provided, comprising:

[0016] A first acquisition module is used to acquire a first sequence and a second sequence. The first sequence includes a local map of a local area corresponding to a first mobile device at each of the multiple time points. The second sequence includes obstacle information around the first mobile device at each of the multiple time points.

[0017] The first generation module is used to generate obstacle prediction data corresponding to multiple prediction tasks based on the first sequence and the second sequence obtained by the first acquisition module, via a multi-task prediction network.

[0018] The first determining module is used to determine the task loss value corresponding to each of the plurality of prediction tasks based on the obstacle prediction data corresponding to the prediction task generated by the first generation model and the corresponding loss calculation method.

[0019] The second determining module is used to determine a multi-task loss value based on the task loss values ​​corresponding to each of the multiple prediction tasks determined by the first determining module.

[0020] The training module is used to train the multi-task prediction network based on the multi-task loss value determined by the second determining module.

[0021] The third determining module is used to determine the multi-task prediction network as a multi-task prediction model in response to the multi-task prediction network trained by the training module meeting the preset training termination condition.

[0022] According to another aspect of the present disclosure, a control device for a mobile device is provided, comprising:

[0023] The second acquisition module is used to acquire a third sequence and a fourth sequence. The third sequence includes a local map of a local area corresponding to the second mobile device at each of the multiple time points. The second sequence includes obstacle information around the second mobile device at each of the multiple time points.

[0024] The second generation module is used to generate obstacle prediction data corresponding to multiple prediction tasks based on the third sequence and the fourth sequence obtained by the second acquisition module, through a multi-task prediction model.

[0025] The control module is used to control the movement of the second mobile device based on the obstacle prediction data corresponding to each of the multiple prediction tasks generated by the second generation module.

[0026] According to another aspect of the present disclosure, a computer-readable storage medium is provided, the storage medium storing a computer program for executing the above-described multi-task prediction model training method or the above-described mobile device control method.

[0027] According to another aspect of the present disclosure, an electronic device is provided, the electronic device comprising:

[0028] processor;

[0029] Memory used to store the processor's executable instructions;

[0030] The processor is configured to read the executable instructions from the memory and execute the instructions to implement the multi-task prediction model training method or the mobile device control method described above.

[0031] Based on the multi-task prediction model training method, mobile device control method, apparatus, computer-readable storage medium, and electronic device provided in the above embodiments of this disclosure, during the model training phase, obstacle prediction data corresponding to multiple prediction tasks can be generated via a multi-task prediction network based on a first sequence composed of local maps corresponding to multiple time points and a second sequence composed of obstacle information corresponding to multiple time points. According to the loss calculation method corresponding to each of the multiple prediction tasks, task loss values ​​corresponding to each of the multiple prediction tasks can be obtained from the obstacle prediction data. Based on the task loss values ​​corresponding to each of the multiple prediction tasks, a multi-task loss value can be determined. By using the multi-task loss value to train the multi-task prediction network, a multi-task prediction model employing a multi-task framework can be obtained. Thus, during the model usage phase, through the execution of multiple prediction tasks, the multi-task prediction model can predict obstacle-related information from different dimensions. Using the prediction results from different dimensions for the driving control of the mobile device enriches the data referenced for driving control, and the prediction results from different dimensions can complement each other, thereby improving the reliability of autonomous driving.

[0032] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0033] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0034] Figure 1 This is a flowchart illustrating a multi-task prediction model training method provided in an exemplary embodiment of this disclosure.

[0035] Figure 2 This is a flowchart illustrating a multi-task prediction model training method provided in another exemplary embodiment of this disclosure.

[0036] Figure 3 This is a flowchart illustrating a multi-task prediction model training method provided in another exemplary embodiment of this disclosure.

[0037] Figure 4 This is a flowchart illustrating a multi-task prediction model training method provided in yet another exemplary embodiment of this disclosure.

[0038] Figure 5 This is a flowchart illustrating a multi-task prediction model training method provided in yet another exemplary embodiment of this disclosure.

[0039] Figure 6 This is a flowchart illustrating a multi-task prediction model training method provided in yet another exemplary embodiment of this disclosure.

[0040] Figure 7 This is a flowchart illustrating a multi-task prediction model training method provided in yet another exemplary embodiment of this disclosure.

[0041] Figure 8 This is a flowchart illustrating a control method for a mobile device provided in an exemplary embodiment of this disclosure.

[0042] Figure 9 This is a flowchart illustrating a mobile device control method provided in another exemplary embodiment of this disclosure.

[0043] Figure 10 This is a schematic diagram of the structure of a multi-task prediction model training device provided in an exemplary embodiment of this disclosure.

[0044] Figure 11 This is a schematic diagram of the structure of a multi-task prediction model training device provided in another exemplary embodiment of this disclosure.

[0045] Figure 12 This is a schematic diagram of the structure of a multi-task prediction model training apparatus provided in another exemplary embodiment of the present disclosure.

[0046] Figure 13 This is a schematic diagram of the structure of a multi-task prediction model training device provided in yet another exemplary embodiment of this disclosure.

[0047] Figure 14 This is a schematic diagram of the structure of a multi-task prediction model training device provided in yet another exemplary embodiment of this disclosure.

[0048] Figure 15 This is a schematic diagram of the structure of a multi-task prediction model training device provided in yet another exemplary embodiment of this disclosure.

[0049] Figure 16 This is a schematic diagram of the structure of a control device for a mobile device provided in an exemplary embodiment of this disclosure.

[0050] Figure 17 This is a schematic diagram of the structure of a control device for a mobile device provided in another exemplary embodiment of this disclosure.

[0051] Figure 18 This is a structural diagram of an electronic device provided in an exemplary embodiment of this disclosure. Detailed Implementation

[0052] Hereinafter, exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present disclosure, and not all embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described herein.

[0053] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0054] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0055] It should also be understood that in the embodiments disclosed herein, "a plurality of" may refer to two or more, and "at least one" may refer to one, two or more.

[0056] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0057] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship.

[0058] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0059] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0060] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0061] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0062] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0063] The embodiments disclosed herein can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.

[0064] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are executed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0065] Application Overview

[0066] The autonomous driving prediction system is an important component of realizing autonomous driving. Based on data collected by sensors such as cameras and radar installed in the vehicle, the autonomous driving prediction system can predict the trajectory of obstacles on the road, and the trajectory prediction results can be used for vehicle driving control.

[0067] In the process of realizing this disclosure, the inventors discovered that trajectory prediction is relatively reliable in short-term prediction, but the error is very large in medium and long-term prediction. Therefore, it is difficult to guarantee the reliability of autonomous driving when the trajectory prediction results are used for vehicle driving control.

[0068] Exemplary methods

[0069] Figure 1 This is a flowchart illustrating a multi-task prediction model training method provided in an exemplary embodiment of this disclosure. Figure 1 The method shown includes steps 110, 120, 130, 140, 150 and 160, which are explained below.

[0070] Step 110: Obtain a first sequence and a second sequence. The first sequence includes: a local map of the local area corresponding to the first mobile device at each of the multiple time points. The second sequence includes: obstacle information around the first mobile device at each of the multiple time points.

[0071] It should be noted that the mobile devices involved in the embodiments of this disclosure can be vehicles, trains, or other devices with mobility functions. For ease of understanding, the embodiments of this disclosure are all described using the case of a vehicle as the mobile device.

[0072] It should be noted that the multiple moments involved in the embodiments of this disclosure can be represented as N moments; where N can be 5, 8, 10, 15 or other values, which will not be listed here.

[0073] At each of the N time points, the sensors installed on the first mobile device can collect data to obtain N sets of collected data corresponding to each of the N time points.

[0074] Optionally, the sensors installed on the first mobile device include, but are not limited to, cameras, radar, Global Positioning System (GPS), Inertial Measurement Unit (IMU), wheel speedometers, etc.

[0075] After obtaining the data set corresponding to any given moment, the GPS-collected positioning information can be extracted from the data set corresponding to that moment. Then, a map area with a preset size, centered on the location corresponding to the positioning information, can be extracted from the pre-constructed high-precision map. The extracted map area can then be used as the local map corresponding to that moment.

[0076] After obtaining the data set corresponding to any given moment, an obstacle recognition algorithm can be used to process the data set corresponding to that moment in order to obtain obstacle information around the first mobile device. The obstacle information includes, but is not limited to, obstacle position, obstacle speed, obstacle orientation, obstacle distance (which can be the distance between the obstacle and the first mobile device).

[0077] The above describes the method for obtaining the local map and obstacle information corresponding to any given time. Following this method, for each of the N time points, the corresponding local map and obstacle information can be determined separately, resulting in N local maps and N obstacle information points corresponding one-to-one with the N time points. Arranging the N local maps in chronological order according to their corresponding time points forms the first sequence in step 110; arranging the N obstacle information points in chronological order according to their corresponding time points forms the second sequence in step 110.

[0078] It should be noted that location information can be obtained not only from GPS, but also from information collected by wheel speedometers or other sensors; local maps can be obtained not only by cropping high-precision maps, but also by using Simultaneous Localization and Mapping (SLAM) technology to build them in real time.

[0079] Step 120: Based on the first sequence and the second sequence, obstacle prediction data corresponding to each of the multiple prediction tasks is generated via a multi-task prediction network.

[0080] It should be noted that the multi-task prediction network can be a prediction network to be trained. The multi-task prediction network can have multiple prediction tasks, which can be represented as M prediction tasks. Each of the M prediction tasks can be an obstacle prediction task related to autonomous driving. M can be 2, 3, 4 or other values, which will not be listed here.

[0081] In step 120, the first sequence and the second sequence can be provided as input data to the multi-task prediction network, which can then perform calculations to generate M obstacle prediction data corresponding to M prediction tasks.

[0082] Of course, in step 120, a certain fusion algorithm can also be used to fuse the information carried by the first sequence and the information carried by the second sequence, and the fusion result is provided as input data to the multi-task prediction network. The multi-task prediction network can then perform calculations to generate M obstacle prediction data that correspond one-to-one with the M prediction tasks.

[0083] Step 130: For each of the multiple prediction tasks, determine the task loss value corresponding to the prediction task based on the obstacle prediction data and the corresponding loss calculation method.

[0084] Optionally, loss calculation methods can be pre-defined for each of the M prediction tasks; the loss calculation methods for different prediction tasks can be the same or different; the loss calculation method for each prediction task can be represented by a loss function.

[0085] In step 130, for each of the M prediction tasks, a loss function can be determined to characterize the loss calculation method corresponding to the prediction task. The obstacle prediction data corresponding to the prediction task is substituted into the loss function for calculation, and the calculation result can be used as the task loss value corresponding to the prediction task. In this way, M task loss values ​​corresponding to the M prediction tasks can be obtained one by one.

[0086] Step 140: Determine the multi-task loss value based on the task loss values ​​corresponding to each of the multiple prediction tasks.

[0087] In step 140, the loss values ​​of the M tasks corresponding to the M prediction tasks can be summed, and the summation result can be used as the multi-task loss value; or, weights can be assigned to the M prediction tasks respectively to obtain M weights corresponding to the M prediction tasks, and the M loss values ​​corresponding to the M prediction tasks can be weighted using the obtained M weights (e.g., weighted summation, weighted average, etc.), and the weighted result can be used as the multi-task loss value.

[0088] Step 150: Train the multi-task prediction network based on the multi-task loss value.

[0089] In step 150, referring to the multi-task loss value, gradient descent (such as stochastic gradient descent, steepest gradient descent, etc.) can be used to update the parameters of the multi-task prediction network to minimize the multi-task loss value, thereby optimizing the parameters of the multi-task prediction network and thus achieving the training of the multi-task prediction network.

[0090] Step 160: In response to the trained multi-task prediction network meeting the preset training termination condition, the trained multi-task prediction network is determined as a multi-task prediction model.

[0091] It should be noted that when training a multi-task prediction network, a large amount of sample data can be used. Each sample data includes a first sequence and a second sequence. In this way, for each sample data, steps 110 to 150 above can be executed. The process of executing steps 110 to 150 above for each sample data can be regarded as an iterative process.

[0092] After several iterations, if the trained multi-task prediction network is found to have converged at a certain point, it can be determined that the trained multi-task prediction network meets the preset training termination condition. At this time, the trained multi-task prediction network can be directly identified as a multi-task prediction model.

[0093] Of course, the preset training termination condition is not limited to this. For example, the multi-task prediction network after training can be determined to meet the preset training termination condition when the number of iterations reaches a preset number.

[0094] Based on the multi-task prediction model training method provided in the above embodiments of this disclosure, during the model training phase, obstacle prediction data corresponding to multiple prediction tasks can be generated via a multi-task prediction network based on a first sequence composed of local maps corresponding to multiple time points and a second sequence composed of obstacle information corresponding to multiple time points. According to the loss calculation method corresponding to each of the multiple prediction tasks, task loss values ​​corresponding to each of the multiple prediction tasks can be obtained from the obstacle prediction data. Based on the task loss values ​​corresponding to each of the multiple prediction tasks, a multi-task loss value can be determined. By using the multi-task loss value to train the multi-task prediction network, a multi-task prediction model employing a multi-task framework can be obtained. Thus, during the model usage phase, through the execution of multiple prediction tasks, the multi-task prediction model can predict obstacle-related information from different dimensions. Using the prediction results from different dimensions for the driving control of the mobile device enriches the data referenced for driving control, and the prediction results from different dimensions complement each other, thereby improving the reliability of autonomous driving.

[0095] exist Figure 1 Based on the illustrated embodiments, as Figure 2 As shown, step 120 includes steps 1201, 1203, 1205, 1207, and 1209.

[0096] Step 1201: Based on the second sequence, determine the obstacle distribution map corresponding to each of the multiple local maps in the first sequence.

[0097] Optionally, the first sequence may include: M local maps arranged in sequence, the M local maps may correspond to the same map coordinate system; the second sequence may include: N obstacle information items arranged in sequence, the N obstacle information items may correspond to the same image coordinate system; the transformation relationship between the map coordinate system and the image coordinate system may be predetermined, and the transformation relationship may be in the form of a transformation matrix.

[0098] In step 1201, for any obstacle information among the N obstacle information sets, the aforementioned transformation relationship can be used to transform the obstacle information from the image coordinate system to the map coordinate system, thereby obtaining an obstacle distribution map in the map coordinate system. This obstacle distribution map can be used to characterize the obstacle distribution in the local map at the same time corresponding to the obstacle information, for example, to characterize which locations in the local map have obstacles. In this way, an obstacle distribution map can be obtained corresponding to the N local map sets. Figure 1 A distribution map of N obstacles corresponding to one obstacle.

[0099] Optionally, the dimensions of each local map and each obstacle distribution map can be consistent. For example, the height and width of each local map and each obstacle distribution map can be preset. The preset height can be represented as H1 and the preset width can be represented as W1.

[0100] Step 1203: Based on the first sequence and the obstacle distribution maps corresponding to the multiple local maps in the first sequence, determine the first feature vector.

[0101] In one alternative implementation, step 1203 includes:

[0102] The first feature map is obtained by overlaying multiple local maps in the first sequence and the obstacle distribution maps corresponding to each local map along the channel direction.

[0103] The first feature map is downsampled to obtain the second feature map;

[0104] The second feature map is converted into the first feature vector.

[0105] Assuming that the height of each local map and each obstacle distribution map is H1 and the width is W1, then by superimposing N local maps and N obstacle distribution maps along the channel direction, the height of the first feature map can be H1, the width can be W1, and the number of dimensions can be 2N.

[0106] Optionally, the N local maps can be located in the first N dimensions of the first feature map, and the N obstacle distribution maps can be located in the last N dimensions of the first feature map; or, the N obstacle distribution maps can be located in the first N dimensions of the first feature map, and the N local maps can be located in the last N dimensions of the first feature map; or, the N local maps and the N obstacle distribution maps can be distributed alternately in the dimensional directions of the first feature map, for example, according to the rule of "local map - obstacle distribution map - local map - obstacle distribution map".

[0107] A multi-task prediction network may include: a downsampling layer, which processes the first feature map to achieve dimensionality reduction, thereby obtaining a second feature map. The height of the second feature map can be H1, the width can be W1, and the number of dimensions can be 1.

[0108] The multi-task prediction network may also include: a fully connected layer, which processes the second feature map to form a feature sequence of a certain length. This feature sequence can be in the form of a one-dimensional vector and can be used as the first feature vector.

[0109] This implementation method, through simple processing methods such as overlay processing and downsampling processing, can efficiently and reliably achieve the fusion of the information carried by the first sequence and the information carried by the N obstacle distribution maps, so as to obtain a first feature vector carrying the fusion result.

[0110] Of course, in step 1203, other information fusion algorithms can also be used to achieve the fusion of the information carried by the first sequence and the information carried by the N obstacle distribution maps.

[0111] Step 1205: Determine the second feature vector based on the second sequence.

[0112] In step 1205, the information of the N obstacles in the second sequence can be converted into feature vectors respectively, and then these feature vectors can be concatenated using a feature concatenation algorithm. The concatenation result can be used as the second feature vector. Alternatively, the second sequence can be directly converted into a third feature map (the conversion method can refer to the method of obtaining the first feature map above), and then the third feature map can be converted into a feature vector, which can be used as the second feature vector.

[0113] Optionally, the second feature vector can be a one-dimensional vector, and the length of the second feature vector can be the same as that of the first feature vector.

[0114] Step 1207: Concatenate the first feature vector with the second feature vector to obtain the third feature vector.

[0115] In step 1207, a feature concatenation algorithm can be used to concatenate the first feature vector and the second feature vector, and the concatenation result can be used as the third feature vector.

[0116] Step 1209: Based on the third feature vector, obstacle prediction data corresponding to each of the multiple prediction tasks is generated through a multi-task prediction network.

[0117] Optionally, the multi-task prediction network may include an information fusion part and a prediction part. The information fusion part may include the downsampling layer and the fully connected layer mentioned above. The information fusion part may perform steps 1201 to 1207 above to obtain a third feature vector and provide the third feature vector to the prediction part. The prediction part may predict obstacle-related information from different dimensions based on the third feature vector to generate M obstacle data corresponding to M prediction tasks one by one.

[0118] In the embodiments of this disclosure, a feature vector of a certain length (i.e., the first feature vector) can be encoded based on the first sequence and the second sequence. At the same time, the original feature vector (i.e., the second feature vector) can also be obtained. By fusing and splicing the encoded feature vector with the original feature vector, a third feature vector that can effectively reflect the information carried by the first sequence and the information carried by the second sequence can be generated. In this way, the multi-task prediction network can refer to the environmental information, obstacle information, etc. around the first mobile device to make predictions from different dimensions, thereby obtaining rich prediction results.

[0119] In an optional example, the prediction component of the multi-task prediction network may include: a trajectory prediction network, a behavior prediction network, and a validity prediction network; the multiple prediction tasks may include: a trajectory prediction task, a behavior prediction task, and a validity prediction task; wherein, the trajectory prediction network can be used to perform the trajectory prediction task, the behavior prediction network can be used to perform the behavior prediction task, and the validity prediction network can be used to perform the validity prediction task.

[0120] Alternatively, the trajectory prediction network can also be called the trajectory prediction head, the behavior prediction network can also be called the behavior prediction head, and the validity prediction network can also be called the validity prediction head.

[0121] Optionally, when calculating the task loss value, the loss calculation methods for trajectory prediction task, behavior prediction task, and effectiveness prediction task can be different.

[0122] exist Figure 1 Based on the illustrated embodiments, as Figure 3 As shown, step 120 includes steps 1211, 1213 and 1215.

[0123] Step 1211: Based on the first sequence and the second sequence, obstacle prediction data corresponding to the trajectory prediction task in the multi-task prediction network is generated.

[0124] After the information fusion part generates the third feature vector based on the first and second sequences, it can provide the third feature vector to the trajectory prediction network. The trajectory prediction network can then perform calculations to generate obstacle prediction data corresponding to the trajectory prediction task.

[0125] Optionally, the obstacle data corresponding to the trajectory prediction task may include: the predicted trajectory of obstacles around the first mobile device and the confidence level corresponding to the predicted trajectory; wherein, the confidence level corresponding to the predicted trajectory can be used to characterize the reliability of the predicted trajectory.

[0126] Step 1213: Based on the first sequence and the second sequence, obstacle prediction data corresponding to the behavior prediction task in multiple prediction tasks is generated through the behavior prediction network in the multi-task prediction network.

[0127] After the information fusion part generates a third feature vector based on the first and second sequences, it can provide the third feature vector to the behavior prediction network. The behavior prediction network can then perform calculations to generate obstacle data corresponding to the behavior prediction task.

[0128] In one alternative implementation, step 1213 includes at least one of the following:

[0129] Based on the first sequence and the second sequence, obstacle prediction data corresponding to the variable speed behavior prediction task in multiple prediction tasks is generated through the variable speed behavior prediction network in the multi-task prediction network.

[0130] Based on the first and second sequences, obstacle prediction data corresponding to the lane change behavior prediction task in the multi-task prediction network is generated.

[0131] Here, there can be two behavior prediction networks: a speed change behavior prediction network and a lane change behavior prediction network. Correspondingly, there can also be two behavior prediction tasks: a speed change behavior prediction task and a lane change behavior prediction task. The speed change behavior prediction network can be used to perform the speed change behavior prediction task, and the lane change behavior prediction network can be used to perform the lane change behavior prediction task.

[0132] Optionally, the obstacle prediction data corresponding to the variable speed behavior prediction task may include: probability vectors corresponding to at least one variable speed behavior; wherein, at least one variable speed behavior may include at least one of the following: constant speed behavior, acceleration behavior, and deceleration behavior; the probability vector corresponding to any variable speed behavior may be used to represent the probability that obstacles around the first mobile device will undergo that variable speed behavior.

[0133] Optionally, the obstacle data corresponding to the lane change behavior prediction task may include: probability vectors corresponding to at least one lane change behavior; wherein, at least one lane change behavior may include at least one of the following: going straight, changing lanes to the left, and changing lanes to the right; the probability vector corresponding to any lane change behavior may be used to represent the probability of the obstacle around the first mobile device causing the lane change behavior.

[0134] Optionally, when calculating the task loss value, the loss calculation method for the variable speed behavior prediction task and the loss calculation method for the lane change behavior prediction task can be the same.

[0135] In this implementation, since the model training phase involves two behavior prediction tasks, the multi-task prediction model can simultaneously predict the speed change behavior and lane change behavior of obstacles during the model usage phase. The prediction results of both speed change behavior and lane change behavior are used for driving control, which helps to improve the reliability of autonomous driving.

[0136] In practice, the number of behavior prediction networks can be as small as one, such as a speed change behavior prediction network or a lane change behavior prediction network, which is also feasible.

[0137] Step 1215: Based on the first sequence and the second sequence, obstacle prediction data corresponding to the effectiveness prediction task in multiple prediction tasks is generated through the effectiveness prediction network in the multi-task prediction network.

[0138] After the information fusion part generates the third feature vector based on the first and second sequences, it can provide the third feature vector to the effectiveness prediction network. The effectiveness prediction network can then perform calculations to generate obstacle prediction data corresponding to the effectiveness prediction task.

[0139] Optionally, the obstacle prediction data corresponding to the validity prediction task may include: a confidence level indicating whether the obstacles around the first mobile device are valid.

[0140] In the embodiments of this disclosure, the multi-task prediction network includes a trajectory prediction network for performing trajectory prediction tasks, a behavior prediction network for performing behavior prediction tasks, and an effectiveness prediction network for performing effectiveness prediction tasks. In this way, during the model usage phase, the multi-task prediction model can simultaneously and reliably predict the trajectory, behavior, and effectiveness of obstacles, and use the prediction results of these dimensions for driving control, which is beneficial to improving the reliability of autonomous driving.

[0141] In an optional example, the obstacle prediction data corresponding to the trajectory prediction task includes: multiple predicted trajectories of the target obstacle around the first mobile device, the confidence level of each of the multiple predicted trajectories, and the probability distribution parameters of each of the multiple predicted trajectories. The multiple predicted trajectories correspond to multiple preset trajectories of the target obstacle.

[0142] It should be noted that the target obstacle can be any obstacle around the first mobile device. Since the calculation methods for each obstacle around the first mobile device are similar, this article will only provide a detailed introduction to the calculation methods for the target obstacle.

[0143] For ease of description, the multiple predicted trajectories involved in the embodiments of this disclosure can be referred to as K predicted trajectories. Correspondingly, the multiple preset trajectories involved in the embodiments of this disclosure can be referred to as K preset trajectories.

[0144] Optionally, the probability distribution parameters corresponding to any trajectory can be Gaussian distribution parameters, such as mean, variance, etc.; there can be a one-to-one correspondence between the K predicted trajectories and the K preset trajectories.

[0145] exist Figure 3 Based on the illustrated embodiments, as Figure 4 As shown, step 130 includes steps 1301, 1303, 1305, 1306, 1307, 1309 and 1311.

[0146] Step 1301: Determine the actual trajectory of the target obstacle within a first preset time period after multiple moments.

[0147] Optionally, the first preset time period after multiple moments can be 3 seconds, 5 seconds, 10 seconds, 15 seconds, etc. after the N moments mentioned above, and will not be listed one by one here.

[0148] It should be noted that during the model training phase, obstacle-related information can be pre-annotated, such as the actual trajectory and behavior of the obstacles. In this way, in step 1301, based on the annotated information, the actual trajectory of the target obstacle can be determined efficiently and reliably.

[0149] Step 1303: Select the first predicted trajectory with the highest confidence level from multiple predicted trajectories.

[0150] In step 1303, the confidence scores of each pair of the K confidence scores that correspond one-to-one with the K predicted trajectories can be compared to select the confidence score with the largest value from the K confidence scores. The predicted trajectory corresponding to this confidence score can then be used as the first predicted trajectory.

[0151] Step 1305: Select the target preset trajectory that is closest to the real trajectory from multiple preset trajectories.

[0152] In step 1305, for each of the K preset trajectories, the similarity can be calculated with the real trajectory to obtain K similarity values ​​that correspond one-to-one with the K preset trajectories. The preset trajectory with the largest similarity value is selected from the K similarity values ​​and can be used as the target preset trajectory.

[0153] Step 1306: Select the second predicted trajectory that corresponds to the target preset trajectory from multiple predicted trajectories.

[0154] Since there is a one-to-one correspondence between the K predicted trajectories and the K preset trajectories, the predicted trajectory corresponding to the target preset trajectory can be determined, and this predicted trajectory can be used as the second predicted trajectory.

[0155] Step 1307: Determine the first loss value based on the probability distribution parameters corresponding to the first predicted trajectory and the probability distribution parameters corresponding to the actual trajectory.

[0156] Optionally, during the model training phase, when annotating obstacle-related information, the corresponding probability distribution parameters can be determined for the real trajectory, and the probability distribution parameters corresponding to the real trajectory can be stored.

[0157] In step 1307, the probability distribution parameters corresponding to the first predicted trajectory can be extracted from the obstacle prediction data corresponding to the trajectory prediction task, the probability distribution parameters corresponding to the stored real trajectory can be obtained, and the difference between the two probability distribution parameters can be evaluated by calculating the relative entropy (Kullback–Leibler divergence, KL divergence) for the two probability distribution parameters, and the first loss value can be determined based on the evaluated difference.

[0158] Optionally, the first loss value can be positively correlated with the assessed difference. Assuming the assessed difference is represented by a numerical value, this value can be directly used as the first loss value; or, the value can be logarithmically calculated with the base e of the natural number, and the result can be used as the first loss value; or, the value can be multiplied by a preset coefficient greater than 0, and the result can be used as the first loss value.

[0159] Step 1309: Determine the second loss value based on the confidence level corresponding to the second predicted trajectory.

[0160] Optionally, the second loss value and the confidence level corresponding to the second predicted trajectory can be negatively correlated. For example, the negative of the confidence level corresponding to the second predicted trajectory can be used as the second loss value; or, the confidence level corresponding to the second predicted trajectory can be logarithmically calculated with the base e of a natural number, and the negative of the result can be used as the second loss value.

[0161] Step 1311: Based on the first loss value and the second loss value, determine the task loss value corresponding to the trajectory prediction task.

[0162] In step 1311, the first loss value and the second loss value can be summed and the summed result can be used as the task loss value corresponding to the trajectory prediction task; or, the first loss value and the second loss value can be assigned weights respectively, and the first loss value and the second loss value can be weighted (e.g., weighted summation or weighted average) using the assigned weights, and the weighted result can be used as the task loss value corresponding to the trajectory prediction task.

[0163] It should be noted that the first loss value mentioned above can be considered as the trajectory regression loss, and the second loss value mentioned above can be considered as the trajectory confidence loss. Therefore, the task loss value corresponding to the trajectory prediction task = trajectory regression loss + trajectory confidence loss.

[0164] In the embodiments of this disclosure, during the model training phase, the trajectory regression loss and trajectory confidence loss can be calculated efficiently and reliably based on the obstacle prediction data corresponding to the trajectory prediction task. Using the trajectory regression loss and trajectory confidence loss, the task loss value corresponding to the trajectory prediction task can be calculated efficiently and reliably. This task loss value can then be used to train the multi-task prediction network. Furthermore, by utilizing the trajectory regression loss, the probability distribution parameters corresponding to the predicted trajectory with the highest confidence generated by the trajectory prediction network when performing the trajectory prediction task are made as close as possible to the real trajectory. By utilizing the trajectory confidence loss, the confidence of the predicted trajectory that is closest to the real trajectory when performing the trajectory prediction task is maximized. Therefore, the embodiments of this disclosure are beneficial in ensuring the accuracy and reliability of the prediction results generated by the trajectory prediction network during the model usage phase.

[0165] In one optional example, the obstacle prediction data corresponding to the behavior prediction task includes: multiple predicted probability values ​​of target obstacles around the first mobile device corresponding to various preset behaviors.

[0166] For ease of description, the various preset behaviors involved in the embodiments of this disclosure can be referred to as R preset behaviors. Correspondingly, the multiple predicted probability values ​​of the target obstacle corresponding to the various preset behaviors can be referred to as R probability values. There can be a one-to-one correspondence between the R probability values ​​and the R preset behaviors.

[0167] exist Figure 3 Based on the illustrated embodiments, as Figure 5 As shown, step 130 includes steps 1313, 1315, 1317 and 1319.

[0168] Step 1313: Determine the actual behavior of the target obstacle within a second preset time period after multiple moments.

[0169] Optionally, the second preset time period and the first preset time period mentioned above can be the same time period.

[0170] It should be noted that during the model training phase, obstacle-related information can be pre-annotated, such as the actual trajectory and behavior of the obstacles. In this way, in step 1313, based on the annotated information, the actual behavior of the target obstacle can be determined efficiently and reliably.

[0171] Step 1315: Select the target predicted probability value that matches the preset behavior and the actual behavior from multiple predicted probability values.

[0172] It should be noted that the actual behavior can be one of the R preset behaviors. In step 1315, the prediction probability value corresponding to the preset behavior can be selected from the R prediction probability values, and this prediction probability value can be used as the target prediction probability value.

[0173] Step 1317: Based on multiple predicted probability values, determine the normalized value corresponding to the target predicted probability value.

[0174] Suppose the target prediction probability value among R prediction probability values ​​is represented as x. i Then, the normalized value G corresponding to the predicted probability value of the target can be calculated using the following formula:

[0175]

[0176] Where exp() represents an exponential function with the natural number e as the base, x i Let x represent the i-th probability value among R predicted probability values. j Let j represent the j-th probability value among R predicted probability values.

[0177] Of course, the method of calculating the normalized value corresponding to the target prediction probability value is not limited to this. Those skilled in the art can also adopt other feasible normalization methods according to actual needs to achieve the normalization processing of the target prediction probability value, thereby obtaining the normalized value corresponding to the target prediction probability value.

[0178] Step 1319: Determine the task loss value corresponding to the behavior prediction task based on the normalized numerical value.

[0179] In step 1319, the normalized value can be directly used as the task loss value corresponding to the behavior prediction task; or, the normalized value can be logarithmically calculated with the base e as the natural number, and the result can be used as the task loss value corresponding to the behavior prediction task.

[0180] In the embodiments of this disclosure, by filtering the target prediction probability values ​​that match the actual behavior and then combining them with normalization processing, the task loss value corresponding to the behavior prediction task can be determined efficiently and reliably.

[0181] It should be noted that when there are two behavior prediction tasks, namely the speed change behavior prediction task and the lane change behavior prediction task, the task loss value corresponding to the speed change behavior prediction task and the task loss value corresponding to the lane change behavior prediction task can be calculated separately according to the above method.

[0182] In one optional example, the obstacle prediction data for the validity prediction task includes: the confidence level of the target obstacles around the first mobile device;

[0183] exist Figure 3 Based on the illustrated embodiments, as Figure 6 As shown, step 130 includes steps 1321, 1323, 1325 and 1327.

[0184] Step 1321: Map the confidence level of the target obstacle to a specified numerical range to obtain the mapped value.

[0185] Optionally, the specified numerical range can be (0, 1). Of course, the specified numerical range can also be (0, 5), (0, 10) or other numerical ranges. For ease of understanding, the embodiments of this disclosure are all illustrated with the case where the specified numerical range is (0, 1).

[0186] Step 1323: Determine the obstacle attributes of the target obstacle based on the mapping value.

[0187] Optionally, after mapping the confidence level of the target obstacle to a specified numerical range, the mapped value can be compared with a preset value (e.g., 0.6, 0.7, or other set values). If the mapped value is greater than the preset value, the obstacle attribute of the target obstacle can be determined to be a valid attribute. If the mapped value is less than or equal to the preset value, the obstacle attribute of the target obstacle can be determined to be an invalid attribute.

[0188] In practice, the ratio of the mapped value to a preset value can be calculated, and the ratio can be compared with another preset value. The comparison result can be used to determine whether the obstacle attribute of the target obstacle is a valid attribute or an invalid attribute.

[0189] Step 1325: In response to the obstacle attribute being a valid attribute, determine the task loss value corresponding to the validity prediction task based on the mapping value.

[0190] If the obstacle attribute is a valid attribute, the task loss value corresponding to the validity prediction task can be determined by referring to the mapping value in a certain way. The task loss value corresponding to the validity prediction task and the mapping value can be positively correlated.

[0191] Optionally, the mapped value can be directly used as the task loss value corresponding to the effectiveness prediction task; or, the mapped value can be subjected to a logarithmic operation with base e, and the result can be used as the task loss value corresponding to the effectiveness prediction task.

[0192] Step 1327: In response to the obstacle attribute being invalid, determine the task loss value corresponding to the validity prediction task based on the difference between the preset value and the mapped value.

[0193] Optionally, the default value can be 1.

[0194] If the obstacle attribute is a valid attribute, the difference between the preset value and the mapped value can be used to determine the task loss value corresponding to the validity prediction task in a certain way. The task loss value corresponding to the validity prediction task and the mapped value can be negatively correlated.

[0195] Optionally, the difference between the preset value and the mapped value can be directly used as the task loss value corresponding to the validity prediction task; or, the difference between the preset value and the mapped value can be logarithmically calculated with base e, and the result can be used as the task loss value corresponding to the validity prediction task.

[0196] In the embodiments of this disclosure, by mapping the confidence level of the target obstacle to a specified numerical range, the obstacle attribute of the target obstacle can be determined efficiently and reliably based on the obtained mapping value. In this way, for the two cases of the obstacle attribute being a valid attribute and an invalid attribute, appropriate methods can be used for calculation, thereby efficiently and reliably obtaining the task loss value corresponding to the validity prediction task.

[0197] The above describes the calculation methods for various task loss values. Assuming that the task loss value corresponding to the trajectory prediction task is simply referred to as the trajectory loss value, the task loss value corresponding to the behavior prediction task is simply referred to as the behavior loss value, and the task loss value corresponding to the effectiveness prediction task is simply referred to as the effectiveness loss value, then we can have: Multi-task loss value = trajectory loss value + behavior loss value + effectiveness loss value.

[0198] In an optional example, such as Figure 7 As shown, during the model training phase, we can first obtain local maps of historical frames (equivalent to the first sequence mentioned above) and obstacle information of historical frames (equivalent to the second sequence mentioned above).

[0199] Next, the information fusion part of the multi-task prediction network can be used to fuse the information carried by the local map of the historical frame and the information carried by the obstacle information of the historical frame to obtain the third feature vector mentioned above.

[0200] Subsequently, the information fusion section provides the third feature vector to the trajectory prediction network, speed change behavior prediction network, lane change behavior prediction network, and effectiveness prediction network, respectively. The trajectory prediction network can then output multimodal trajectories (corresponding to the multiple predicted trajectories mentioned above) and corresponding confidence scores. The speed change behavior prediction network can output speed change probability vectors, the lane change behavior prediction network can output lane change probability vectors, and the effectiveness prediction network can output confidence scores representing the effectiveness of obstacles. Based on the outputs of the trajectory prediction network, the trajectory loss value can be calculated. Based on the outputs of the speed change and lane change behavior prediction networks, the behavior loss value can be calculated. Based on the output of the effectiveness prediction network, the effectiveness loss value can be calculated. Using the trajectory loss value, behavior loss value, and effectiveness loss value, the multi-task loss value can be calculated. This multi-task loss value can be used to train the multi-task prediction network, thereby obtaining a trained multi-task prediction model.

[0201] Any of the multi-task prediction model training methods provided in this disclosure can be executed by any suitable device with data processing capabilities, including but not limited to: terminal devices and servers. Alternatively, any of the multi-task prediction model training methods provided in this disclosure can be executed by a processor, such as by a processor executing any of the multi-task prediction model training methods mentioned in this disclosure by calling corresponding instructions stored in memory. Further details will not be elaborated below.

[0202] Figure 8 This is a flowchart illustrating a control method for a mobile device provided in an exemplary embodiment of this disclosure. Figure 8 The method shown includes steps 810, 820 and 830, which are explained below.

[0203] Step 810: Obtain the third sequence and the fourth sequence. The third sequence includes: at each of the multiple time points, a local map of the local area corresponding to the second mobile device. The fourth sequence includes: at each of the multiple time points, obstacle information around the second mobile device.

[0204] Step 820: Based on the third and fourth sequences, obstacle prediction data corresponding to each of the multiple prediction tasks is generated through a multi-task prediction model.

[0205] It should be noted that steps 810 to 820 are similar to steps 110 to 120 above. The main difference is that steps 810 to 820 are performed during the model usage phase, while steps 110 to 120 are performed during the model training phase. The specific implementation of steps 810 to 820 can be referred to the relevant introduction of steps 110 to 120 above, and will not be repeated here.

[0206] Step 830: Based on the obstacle prediction data corresponding to each of the multiple prediction tasks, drive control is performed on the second mobile device.

[0207] Optionally, the obstacle data corresponding to each of the multiple prediction tasks may include at least two of the following: obstacle prediction data corresponding to the trajectory prediction task, obstacle prediction data corresponding to the speed change behavior prediction task, obstacle prediction data corresponding to the lane change behavior prediction task, and obstacle prediction data corresponding to the effectiveness prediction task.

[0208] In step 830, by referring to the obstacle prediction data corresponding to each of the multiple prediction tasks, a driving strategy can be reasonably planned for the second mobile device, and the driving control of the second mobile device can be performed based on the planning results.

[0209] In one example, the second mobile device is traveling in the leftmost lane of a three-lane road. Using obstacle prediction data corresponding to the lane change behavior prediction task, the second mobile device predicts that the vehicle in front of it on the right intends to change lanes to the leftmost lane. By controlling the second mobile device, its speed can be reduced to avoid a collision with the vehicle in front of it changing lanes to the left.

[0210] In another example, the second mobile device is traveling in the middle lane of a three-lane road. The second mobile device needs to change lanes to the leftmost lane. Furthermore, the second mobile device uses obstacle prediction data corresponding to the speed change behavior prediction task to predict that the vehicle behind it on the left intends to accelerate. By controlling the second mobile device, the second mobile device can be made to temporarily delay changing lanes to the left to avoid a collision with the vehicle accelerating behind it on the left.

[0211] In the embodiments of this disclosure, during the model usage phase, through the execution of multiple prediction tasks, the multi-task prediction model can predict obstacle-related information from different dimensions. The prediction results from different dimensions are used for the driving control of the mobile device, which helps to enrich the data referenced for the driving control of the mobile device. Furthermore, the prediction results from different dimensions can complement each other, thereby improving the reliability of autonomous driving.

[0212] In one optional example, one of the multiple prediction tasks is the validity prediction task;

[0213] exist Figure 8 Based on the illustrated embodiments, as Figure 9 As shown, step 830 includes steps 8301, 8303, 8305 and 8307.

[0214] Step 8301: Decode the obstacle prediction data corresponding to each of the multiple prediction tasks to obtain the obstacle prediction results.

[0215] In step 8301, the decoding module can be used to decode the obstacle prediction data corresponding to each of the multiple prediction tasks to obtain the obstacle prediction results.

[0216] In one example, the obstacle prediction data corresponding to the trajectory prediction task includes: multiple predicted trajectories of an obstacle around the second mobile device, and the confidence level of each of the multiple predicted trajectories. Then, the predicted trajectory with the highest confidence level can be selected from the multiple predicted trajectories. This predicted trajectory can be used as the final predicted trajectory of the obstacle. The obstacle prediction result can include this predicted trajectory.

[0217] In another example, the obstacle prediction data for the variable speed behavior prediction task includes: a first probability value corresponding to a uniform speed behavior, a second probability value corresponding to an acceleration behavior, and a third probability value corresponding to a deceleration behavior for a certain obstacle around the second mobile device. Then, the probability value with the largest value can be selected from the first probability value, the second probability value, and the third probability value. The variable speed behavior corresponding to this probability value can be used as the final variable speed behavior of the obstacle. The obstacle prediction result can include: this variable speed behavior.

[0218] In another example, the obstacle prediction data corresponding to the validity prediction task includes: the confidence level of a certain obstacle around the second mobile device. The confidence level can be mapped to a specified numerical range, and the obtained mapped value is compared with a preset value (e.g., 0.6, 0.7 or other set values). The obstacle attribute of the obstacle is determined by referring to the comparison result. The obstacle prediction result may include the obstacle attribute.

[0219] Step 8303: Post-process the obstacle prediction results to obtain the post-processed results.

[0220] In step 8303, based on the obstacle prediction results, a driving strategy can be planned for the second mobile device, and the planned driving strategy can be used as a post-processing result.

[0221] Step 8305: Based on the results corresponding to the validity prediction task in the obstacle prediction results, filter the results associated with obstacles whose obstacle attributes are invalid in the post-processing results.

[0222] In the scenario described above, where the second mobile device is traveling in the leftmost lane of a three-lane road system, and the second mobile device uses obstacle prediction data from the lane change prediction task to predict that the vehicle to its right intends to change lanes to the leftmost lane, the driving strategy as a post-processing result could include: deceleration. If the obstacle attribute corresponding to the vehicle to its right in the obstacle prediction result is an invalid attribute, then deceleration in this driving strategy can be filtered out; for example, deceleration can be removed from this driving strategy.

[0223] Corresponding to the scenario described above where the second mobile device is traveling in the middle lane of a three-lane road system, and needs to change lanes to the leftmost lane, while the vehicle behind it on the left intends to accelerate, the driving strategy as a post-processing result could include: temporarily suspending the left lane change. If the obstacle attribute corresponding to the vehicle behind it on the left is invalid in the obstacle prediction result, the temporary suspension of the left lane change in this driving strategy can be filtered out; for example, the temporary suspension of the left lane change can be removed from this driving strategy.

[0224] Step 8307: Based on the filtered post-processing results, drive control is performed on the second mobile device.

[0225] Assuming that deceleration in the driving strategy resulting from post-processing is filtered out in step 8305, the second mobile device can accelerate or maintain a constant speed without deceleration by controlling the second mobile device. Assuming that the postponement of left lane change in the driving strategy resulting from post-processing is filtered out in step 8305, the second mobile device can change lanes to the left normally by controlling the second mobile device.

[0226] In the embodiments of this disclosure, through decoding processing, obstacle prediction results can be obtained efficiently and reliably from the obstacle prediction data corresponding to each of the multiple prediction tasks. After obtaining the post-processing results of the obstacle prediction results, the results corresponding to the valid prediction tasks in the obstacle prediction results can be referenced to distinguish between the valid and invalid parts in the post-processing results, so as to filter out the invalid parts and use only the valid parts for the driving control of the second mobile device. This is beneficial for the reasonable driving control of the second mobile device.

[0227] The control method for any mobile device provided in this disclosure can be executed by any suitable device with data processing capabilities, including but not limited to terminal devices and servers. Alternatively, the control method for any mobile device provided in this disclosure can be executed by a processor, such as by a processor executing the control method for any mobile device mentioned in this disclosure by calling corresponding instructions stored in memory. Further details will not be elaborated below.

[0228] Exemplary device

[0229] Figure 10 This is a schematic diagram of the structure of a multi-task prediction model training device provided in an exemplary embodiment of this disclosure. Figure 10 The apparatus shown includes a first acquisition module 1010, a first generation module 1020, a first determination module 1030, a second determination module 1040, a training module 1050, and a third determination module 1060.

[0230] The first acquisition module 1010 is used to acquire a first sequence and a second sequence. The first sequence includes: at each of multiple times, a local map of a local area corresponding to the first mobile device. The second sequence includes: at each of multiple times, obstacle information around the first mobile device.

[0231] The first generation module 1020 is used to generate obstacle prediction data corresponding to multiple prediction tasks based on the first sequence and the second sequence obtained by the first acquisition module 1010, through a multi-task prediction network.

[0232] The first determining module 1030 is used to determine the task loss value corresponding to each of the multiple prediction tasks based on the obstacle prediction data and the corresponding loss calculation method generated by the first generation model for the prediction task.

[0233] The second determining module 1040 is used to determine the multi-task loss value based on the task loss values ​​corresponding to each of the multiple prediction tasks determined by the first determining module 1030.

[0234] The training module 1050 is used to train the multi-task prediction network based on the multi-task loss value determined by the second determining module 1040.

[0235] The third determining module 1060 is used to determine the trained multi-task prediction network as a multi-task prediction model in response to the training module 1050 meeting the preset training termination conditions.

[0236] In an optional example, Figure 10 Based on the illustrated embodiments, as Figure 11 As shown, the first generation module 1020 includes:

[0237] The first generation submodule 10201 is used to generate obstacle prediction data corresponding to the trajectory prediction task in multiple prediction tasks based on the first sequence and the second sequence obtained by the first acquisition module 1010, through the trajectory prediction network in the multi-task prediction network.

[0238] The second generation submodule 10203 is used to generate obstacle prediction data corresponding to the behavior prediction tasks in multiple prediction tasks based on the first sequence and the second sequence obtained by the first acquisition module 1010, through the behavior prediction network in the multi-task prediction network.

[0239] The third generation submodule 10205 is used to generate obstacle prediction data corresponding to the validity prediction task in multiple prediction tasks based on the first sequence and the second sequence obtained by the first acquisition module 1010, through the validity prediction network in the multi-task prediction network.

[0240] In one optional example, the second generation submodule 10203 includes at least one of the following:

[0241] The first generation unit is used to generate obstacle prediction data corresponding to the variable speed behavior prediction task in multiple prediction tasks based on the first sequence and the second sequence obtained by the first acquisition module 1010, via the variable speed behavior prediction network in the multi-task prediction network.

[0242] The second generation unit is used to generate obstacle prediction data corresponding to the lane change behavior prediction task in multiple prediction tasks based on the first sequence and the second sequence obtained by the first acquisition module 1010, via the lane change behavior prediction network in the multi-task prediction network.

[0243] In an optional example, the obstacle prediction data corresponding to the trajectory prediction task includes: multiple predicted trajectories of target obstacles around the first mobile device, the confidence level of each of the multiple predicted trajectories, and the probability distribution parameters of each of the multiple predicted trajectories. The multiple predicted trajectories correspond to multiple preset trajectories of the target obstacles.

[0244] exist Figure 11 Based on the illustrated embodiments, as Figure 12 As shown, the first determining module 1030 includes:

[0245] The first determining submodule 10301 is used to determine the true trajectory of the target obstacle within a first preset time period after multiple moments;

[0246] The first selection submodule 10303 is used to select the first predicted trajectory with the highest confidence from multiple predicted trajectories;

[0247] The second selection submodule 10305 is used to select, from multiple preset trajectories, the target preset trajectory that is closest to the real trajectory determined by the first determination submodule 10301;

[0248] The third selection submodule 10306 is used to select the second predicted trajectory corresponding to the target preset trajectory from multiple predicted trajectories;

[0249] The second determining submodule 10307 is used to determine a first loss value based on the probability distribution parameters corresponding to the first predicted trajectory selected by the first selection submodule 10303 and the probability distribution parameters corresponding to the real trajectory determined by the first determining submodule 10301.

[0250] The third determining submodule 10309 is used to determine the second loss value based on the confidence level corresponding to the second predicted trajectory selected by the third selection submodule 10306.

[0251] The fourth determining submodule 10311 is used to determine the task loss value corresponding to the trajectory prediction task based on the first loss value determined by the second determining submodule 10307 and the second loss value determined by the third determining submodule 10309.

[0252] In one optional example, the obstacle prediction data corresponding to the behavior prediction task includes: multiple predicted probability values ​​of target obstacles around the first mobile device corresponding to various preset behaviors;

[0253] exist Figure 11 Based on the illustrated embodiments, as Figure 13 As shown, the first determining module 1030 includes:

[0254] The fifth determination submodule 10313 is used to determine the actual behavior of the target obstacle within a second preset time period after multiple moments;

[0255] The fourth selection submodule 10315 is used to select, from multiple predicted probability values, the target predicted probability value that matches the preset behavior and the actual behavior determined by the fifth determination submodule 10313.

[0256] The sixth determining submodule 10317 is used to determine the normalized value corresponding to the target prediction probability value selected by the fourth selection submodule 10315 based on multiple prediction probability values.

[0257] The seventh determination submodule 10319 is used to determine the task loss value corresponding to the behavior prediction task based on the normalized value determined by the sixth determination submodule 10317.

[0258] In one optional example, the obstacle prediction data for the validity prediction task includes: the confidence level of the target obstacles around the first mobile device;

[0259] exist Figure 11 Based on the illustrated embodiments, as Figure 14 As shown, the first determining module 1030 includes:

[0260] The mapping submodule 10321 is used to map the confidence level of the target obstacle to a specified numerical range to obtain the mapped value;

[0261] The eighth determining submodule 10323 is used to determine the obstacle attributes of the target obstacle based on the mapping value obtained by the mapping submodule 10321;

[0262] The ninth determining submodule 10325 is used to determine the task loss value corresponding to the validity prediction task based on the mapping value obtained by the mapping submodule 10321 in response to the obstacle attribute determined by the eighth determining submodule 10323 being a valid attribute.

[0263] The tenth determination submodule 10327 is used to determine the task loss value corresponding to the validity prediction task based on the difference between the preset value and the mapping value obtained by the mapping submodule 10321 in response to the obstacle attribute determined by the eighth determination submodule 10323 being an invalid attribute.

[0264] In an optional example, Figure 10 Based on the illustrated embodiments, as Figure 15 As shown, the first generation module 1020 includes:

[0265] The eleventh determining submodule 10207 is used to determine the obstacle distribution map corresponding to each of the multiple local maps in the first sequence obtained by the first acquisition module 1010 based on the second sequence obtained by the first acquisition module 1010.

[0266] The twelfth determining submodule 10209 is used to determine the first feature vector based on the first sequence obtained by the first acquisition module 1010 and the obstacle distribution map corresponding to each of the multiple local maps in the first sequence determined by the eleventh determining submodule 10207.

[0267] The thirteenth determining submodule 10211 is used to determine the second feature vector based on the second sequence obtained by the first obtaining module 1010;

[0268] The splicing submodule 10213 is used to splice the first feature vector determined by the twelfth determination submodule 10209 with the second feature vector determined by the thirteenth determination submodule 10211 to obtain the third feature vector;

[0269] The fourth generation submodule 10215 is used to generate obstacle prediction data for each of the multiple prediction tasks based on the third feature vector obtained by the splicing submodule 10213 via a multi-task prediction network.

[0270] In one optional example, the twelfth determined submodule 10209 includes:

[0271] The overlay unit is used to overlay multiple local maps in the first sequence acquired by the first acquisition module 1010 and the obstacle distribution maps corresponding to the multiple local maps in the first sequence determined by the eleventh determination submodule 10207 along the channel direction to obtain a first feature map.

[0272] The processing unit is used to perform downsampling processing on the first feature map obtained by the superposition unit to obtain the second feature map;

[0273] The transformation unit is used to transform the second feature map into the first feature vector.

[0274] Figure 16 This is a schematic diagram of the structure of a control device for a mobile device provided in an exemplary embodiment of this disclosure. Figure 16 The apparatus shown includes a second acquisition module 1610, a second generation module 1620, and a control module 1630.

[0275] The second acquisition module 1610 is used to acquire a third sequence and a fourth sequence. The third sequence includes a local map of the local area corresponding to the second mobile device at each of the multiple time points. The second sequence includes obstacle information around the second mobile device at each of the multiple time points.

[0276] The second generation module 1620 is used to generate obstacle prediction data corresponding to multiple prediction tasks based on the third sequence and the fourth sequence obtained by the second acquisition module 1610, through a multi-task prediction model.

[0277] The control module 1630 is used to control the movement of the second mobile device based on the obstacle prediction data corresponding to each of the multiple prediction tasks generated by the second generation module 1620.

[0278] In one optional example, one of the multiple prediction tasks is the validity prediction task;

[0279] exist Figure 16 Based on the illustrated embodiments, as Figure 17 As shown, the control module 1630 includes:

[0280] The decoding submodule 16301 is used to decode the obstacle prediction data corresponding to each of the multiple prediction tasks generated by the second generation module 1620 to obtain the obstacle prediction results.

[0281] The processing submodule 16303 is used to perform post-processing on the obstacle prediction results obtained by the decoding submodule 16301 to obtain the post-processed result.

[0282] The filtering submodule 16305 is used to filter the results associated with obstacles whose obstacle attributes are invalid attributes in the post-processing results obtained by the processing submodule 16303, based on the results corresponding to the validity prediction task in the obstacle prediction results obtained by the decoding submodule 16301.

[0283] The control submodule 16307 is used to control the movement of the second mobile device based on the filtered post-processing result obtained by the filtering submodule 16305.

[0284] Exemplary electronic devices

[0285] Below, for reference Figure 18 This describes an electronic device according to embodiments of the present disclosure. The electronic device may be either or both of a first device and a second device, or a standalone device independent of them, which may communicate with the first device and the second device to receive acquired input signals from them.

[0286] Figure 18 A block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0287] like Figure 18 As shown, the electronic device 1800 includes one or more processors 1810 and memory 1820.

[0288] The processor 1810 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 1800 to perform desired functions.

[0289] The memory 1820 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 1810 may execute the program instructions to implement the multi-task prediction model training method of the various embodiments of the present disclosure described above, or to implement the mobile device control method of the various embodiments of the present disclosure described above. Various contents such as input signals, signal components, and noise components may also be stored in the computer-readable storage medium.

[0290] In one example, the electronic device 1800 may also include an input device 1830 and an output device 1840, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0291] For example, when the electronic device is a first device or a second device, the input device 1830 can be the aforementioned microphone or microphone array for capturing input signals from a sound source. When the electronic device is a standalone device, the input device 1830 can be a communication network connector for receiving the acquired input signals from the first device and the second device.

[0292] In addition, the input device 1830 may also include, for example, a keyboard, a mouse, etc.

[0293] The output device 1840 can output various information to the outside, including determined distance information, direction information, etc. The output device 1840 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0294] Of course, for the sake of simplicity, Figure 18 Only some of the components of the electronic device 1800 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 1800 may include any other suitable components depending on the specific application.

[0295] Exemplary computer program products and computer-readable storage media

[0296] In addition to the methods and devices described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform steps in the multi-task prediction model training method according to various embodiments of this disclosure as described in the "Exemplary Methods" section of this specification, or steps in the control method of a mobile device according to various embodiments of this disclosure.

[0297] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0298] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform steps in the multi-task prediction model training method described in the "Exemplary Methods" section of this specification, or steps in the control method of a mobile device according to various embodiments of this disclosure.

[0299] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0300] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0301] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0302] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0303] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0304] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A method for training a multi-task prediction model, comprising: Obtain a first sequence and a second sequence, wherein the first sequence includes: a local map of a local area corresponding to a first mobile device at each of the multiple time points, and the second sequence includes: obstacle information around the first mobile device at each of the multiple time points; Based on the first sequence and the second sequence, obstacle prediction data corresponding to each of the multiple prediction tasks is generated via a multi-task prediction network; For each of the multiple prediction tasks, the task loss value corresponding to the prediction task is determined based on the obstacle prediction data corresponding to the prediction task and the corresponding loss calculation method. Based on the task loss values ​​corresponding to each of the multiple prediction tasks, determine the multi-task loss value; The multi-task prediction network is trained based on the multi-task loss value; In response to the trained multi-task prediction network meeting the preset training termination condition, the trained multi-task prediction network is determined as a multi-task prediction model.

2. The method according to claim 1, wherein, The step of generating obstacle prediction data for multiple prediction tasks based on the first sequence and the second sequence via a multi-task prediction network includes: Based on the first sequence and the second sequence, obstacle prediction data corresponding to the trajectory prediction task in the multiple prediction tasks is generated via the trajectory prediction network in the multi-task prediction network. Based on the first sequence and the second sequence, obstacle prediction data corresponding to the behavior prediction tasks in the multiple prediction tasks is generated via the behavior prediction network in the multi-task prediction network. Based on the first sequence and the second sequence, obstacle prediction data corresponding to the validity prediction task in the multiple prediction tasks is generated via the validity prediction network in the multi-task prediction network; wherein, the validity prediction network is a network used to predict the confidence level of obstacles.

3. The method according to claim 2, wherein, The step of generating obstacle prediction data corresponding to the behavior prediction tasks among the multiple prediction tasks based on the first sequence and the second sequence via the behavior prediction network in the multi-task prediction network includes at least one of the following: Based on the first sequence and the second sequence, obstacle prediction data corresponding to the variable speed behavior prediction task in the multiple prediction tasks is generated via the variable speed behavior prediction network in the multi-task prediction network. Based on the first sequence and the second sequence, obstacle prediction data corresponding to the lane change behavior prediction task in the multiple prediction tasks is generated via the lane change behavior prediction network in the multi-task prediction network.

4. The method according to claim 2, wherein, The obstacle prediction data corresponding to the trajectory prediction task includes: multiple predicted trajectories of target obstacles around the first mobile device, the confidence level of each of the multiple predicted trajectories, and the probability distribution parameters of each of the multiple predicted trajectories. The multiple predicted trajectories correspond to multiple preset trajectories of the target obstacles. For each of the plurality of prediction tasks, the step of determining the task loss value corresponding to that prediction task based on the obstacle prediction data and the corresponding loss calculation method includes: Determine the true trajectory of the target obstacle within a first preset time period following the plurality of moments; From the multiple predicted trajectories, select the first predicted trajectory with the highest confidence level; From the multiple preset trajectories, select the target preset trajectory that is closest to the actual trajectory; From the multiple predicted trajectories, select the second predicted trajectory that corresponds to the target preset trajectory; Based on the probability distribution parameters corresponding to the first predicted trajectory and the probability distribution parameters corresponding to the real trajectory, a first loss value is determined; The second loss value is determined based on the confidence level corresponding to the second predicted trajectory; Based on the first loss value and the second loss value, the task loss value corresponding to the trajectory prediction task is determined.

5. The method according to claim 2, wherein, The obstacle prediction data corresponding to the behavior prediction task includes: multiple prediction probability values ​​of target obstacles around the first mobile device corresponding to various preset behaviors; For each of the plurality of prediction tasks, the step of determining the task loss value corresponding to that prediction task based on the obstacle prediction data and the corresponding loss calculation method includes: Determine the actual behavior of the target obstacle within a second preset time period following the plurality of moments; From the plurality of predicted probability values, select the target predicted probability value that matches the preset behavior with the actual behavior; Based on the multiple predicted probability values, determine the normalized value corresponding to the target predicted probability value; Based on the normalized value, the task loss value corresponding to the behavior prediction task is determined.

6. The method according to claim 2, wherein, The obstacle prediction data corresponding to the effectiveness prediction task includes: the confidence level of the target obstacles around the first mobile device; For each of the plurality of prediction tasks, the step of determining the task loss value corresponding to that prediction task based on the obstacle prediction data and the corresponding loss calculation method includes: The confidence level of the target obstacle is mapped to a specified numerical range to obtain the mapped value; Based on the mapping value, the obstacle attributes of the target obstacle are determined; In response to the obstacle attribute being a valid attribute, the task loss value corresponding to the validity prediction task is determined based on the mapping value; In response to the obstacle attribute being invalid, the task loss value corresponding to the validity prediction task is determined based on the difference between the preset value and the mapping value.

7. The method according to claim 1, wherein, The step of generating obstacle prediction data for multiple prediction tasks based on the first sequence and the second sequence via a multi-task prediction network includes: Based on the second sequence, determine the obstacle distribution map corresponding to each of the multiple local maps in the first sequence; Based on the first sequence and the obstacle distribution map corresponding to each of the multiple local maps in the first sequence, a first feature vector is determined; Based on the second sequence, determine the second feature vector; The first feature vector is concatenated with the second feature vector to obtain the third feature vector; Based on the third feature vector, obstacle prediction data corresponding to each of the multiple prediction tasks is generated via a multi-task prediction network.

8. The method according to claim 7, wherein, The step of determining the first feature vector based on the first sequence and the obstacle distribution maps corresponding to the multiple local maps in the first sequence includes: The first feature map is obtained by overlaying multiple local maps in the first sequence and the obstacle distribution maps corresponding to each local map along the channel direction. The first feature map is downsampled to obtain the second feature map; The second feature map is converted into the first feature vector.

9. A method for controlling a mobile device, comprising: A third sequence and a fourth sequence are obtained, wherein the third sequence includes a local map of a local area corresponding to the second mobile device at each of the multiple time points, and the fourth sequence includes obstacle information around the second mobile device at each of the multiple time points. Based on the third sequence and the fourth sequence, obstacle prediction data corresponding to each of the multiple prediction tasks are generated through a multi-task prediction model; Based on the obstacle prediction data corresponding to each of the multiple prediction tasks, the second mobile device is driven. The multi-task prediction model is obtained using the multi-task prediction model training method described in any one of claims 1-8.

10. The method according to claim 9, wherein, One of the multiple prediction tasks is the effectiveness prediction task; The step of controlling the second mobile device based on obstacle prediction data corresponding to each of the multiple prediction tasks includes: The obstacle prediction data corresponding to each of the multiple prediction tasks is decoded to obtain the obstacle prediction results; The obstacle prediction results are post-processed to obtain the post-processed result; Based on the results corresponding to the validity prediction task in the obstacle prediction results, the results associated with obstacles whose obstacle attributes are invalid in the post-processing results are filtered. Based on the filtered post-processing results, driving control is performed on the second mobile device.

11. A multi-task prediction model training device, comprising: A first acquisition module is used to acquire a first sequence and a second sequence. The first sequence includes a local map of a local area corresponding to a first mobile device at each of the multiple time points. The second sequence includes obstacle information around the first mobile device at each of the multiple time points. The first generation module is used to generate obstacle prediction data corresponding to multiple prediction tasks based on the first sequence and the second sequence obtained by the first acquisition module, via a multi-task prediction network. The first determining module is used to determine the task loss value corresponding to each of the plurality of prediction tasks based on the obstacle prediction data corresponding to the prediction task generated by the first generation model and the corresponding loss calculation method. The second determining module is used to determine a multi-task loss value based on the task loss values ​​corresponding to each of the multiple prediction tasks determined by the first determining module. The training module is used to train the multi-task prediction network based on the multi-task loss value determined by the second determining module. The third determining module is used to determine the multi-task prediction network as a multi-task prediction model in response to the multi-task prediction network trained by the training module meeting the preset training termination condition.

12. A control device for a mobile device, comprising: The second acquisition module is used to acquire a third sequence and a fourth sequence. The third sequence includes a local map of a local area corresponding to the second mobile device at each of the multiple times. The second sequence includes obstacle information around the second mobile device at each of the multiple times. The second generation module is used to generate obstacle prediction data corresponding to multiple prediction tasks based on the third sequence and the fourth sequence obtained by the second acquisition module, through a multi-task prediction model. The control module is used to control the movement of the second mobile device based on the obstacle prediction data corresponding to each of the multiple prediction tasks generated by the second generation module. The multi-task prediction model is obtained using the multi-task prediction model training method described in any one of claims 1-8.

13. A computer-readable storage medium storing a computer program for executing the multi-task prediction model training method according to any one of claims 1-8, or executing the control method of a mobile device according to any one of claims 9-10.

14. An electronic device, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the multi-task prediction model training method according to any one of claims 1-8, or to execute the mobile device control method according to any one of claims 9-10.

Images