Behavior decision-making method, device, equipment, medium, program product and autonomous vehicle

Abstract

The invention provides a behavior decision-making method, device and equipment, a medium, a program product and an automatic driving vehicle, and can be applied to the technical field of image processing and automatic driving. The method comprises the following steps: when a first image frame collected at the t-th moment is an image with fog, carrying out defogging processing on the image with fog to obtain a defogged image; based on an attention mechanism, state detection is carried out on the defogged image to obtain state information, and the state information is used for representing information of the environment where the target object is located at the t-th moment; based on deep reinforcement learning, behavior decision making is carried out on the state information, an initial decision making result is obtained, and the initial decision making result is used for representing action information needing to be executed by the target object from the t-th moment to the (t + 1)-th moment; and determining the initial decision result as a target decision result when it is determined that the initial decision result meets the first preset condition.

Description

Technical Field

[0001] The present disclosure relates to the fields of image processing and autonomous driving technology, and in particular to behavior decision methods, devices, equipment, media, program products and autonomous driving vehicles. Background Art

[0002] In foggy scenes, natural light will be absorbed or scattered by these suspended water droplets and particles during the propagation process, causing the outline of the captured scene to be blurred, and the image clarity and color saturation to decrease, thus affecting the overall visual effect of the image. Especially when the concentration of haze is high, the information of the real image will be seriously lost, hindering the subsequent application of the image.

[0003] In the process of implementing the present disclosure, it was found that in foggy scenes, the existing defogging methods based on prior and learning mapping have large errors. When decisions need to be made on the driving behavior of the target object, the errors are transmitted, affecting the safe operation of the target object. Summary of the invention

[0004] In view of the above problems, the present disclosure provides a behavior decision-making method, device, equipment, medium, program product and autonomous driving vehicle.

[0005] According to a first aspect of the present disclosure, there is provided a behavior decision method, comprising:

[0006] When the first image frame collected at the time t is a foggy image, defogging the foggy image to obtain a defogged image;

[0007] Based on the attention mechanism, the state of the dehazed image is detected to obtain the state information, where the state information is used to represent the environment information of the target object at the tth moment;

[0008] Based on deep reinforcement learning, a behavior decision is made on the state information to obtain an initial decision result, wherein the initial decision result is used to represent the action information that the target object needs to perform from the tth moment to the t+1th moment; and

[0009] When it is determined that the initial decision result meets the first preset condition, the initial decision result is determined as the target decision result.

[0010] According to an embodiment of the present disclosure, the defogging image includes an image of a lane in front of the target object in the direction of travel;

[0011] Behavioral decision-making methods also include:

[0012] Determine the length of the lane in the lane image based on the lane image ahead of the target object in the direction of travel;

[0013] Based on the initial decision result and the length of the lane, the target object is simulated to obtain a simulation result, wherein the simulation result is used to characterize the driving information of the target object when executing the initial decision result;

[0014] Determining an evaluation value for an initial decision result based on the simulation result and a preset evaluation function; and

[0015] In the case where it is determined that the evaluation value exceeds the threshold, it is determined that the initial decision result meets the first preset condition.

[0016] According to an embodiment of the present disclosure, the initial decision result includes lateral acceleration information and longitudinal acceleration information, and the simulation result includes driving speed information and driving collision situation information when the initial decision result is executed;

[0017] Based on the initial decision result and the length of the lane, the target object is simulated to obtain simulation results, including:

[0018] Generate a simulation execution instruction according to the lateral acceleration information, the longitudinal acceleration information and the speed information of the target object at the tth time, wherein the simulation execution instruction is used to indicate the length of the lane in which the target object is traveling; and

[0019] The driving speed information and driving collision information of the target object during the process of executing the simulation execution instruction are counted.

[0020] According to an embodiment of the present disclosure, the preset evaluation function includes a safety constraint item, a speed evaluation item, a comfort evaluation item, and a detection distance evaluation item;

[0021] Based on the simulation results and the preset evaluation function, an evaluation value for the initial decision result is determined, including:

[0022] Determine the value of the safety constraint item according to the driving collision situation information and the preset safety constraint condition;

[0023] Determine the driving speed information of the target object when it completes executing the simulation execution instruction as the value of the speed evaluation item;

[0024] Determining the value of the comfort evaluation item according to the driving speed information corresponding to the preset time interval during the process of the target object executing the simulation execution instruction;

[0025] determining the length of the lane as a value of the detection distance evaluation item; and

[0026] The value of the safety constraint item, the value of the speed evaluation item, the value of the comfort evaluation item, and the value of the detection distance evaluation item are summed to obtain an evaluation value for the initial decision result.

[0027] According to an embodiment of the present disclosure, when the first image frame collected at time t is a foggy image, defogging the foggy image to obtain the defogged image includes:

[0028] Perform feature extraction on the foggy image to obtain a feature image;

[0029] Performing residual prediction on the feature image to obtain a predicted residual image; and

[0030] The predicted residual image and the feature image are fused to obtain the dehazed image.

[0031] According to an embodiment of the present disclosure, based on deep reinforcement learning, behavior decision is made on state information to obtain an initial decision result, including:

[0032] The state information is input into the policy network in deep reinforcement learning, and an initial decision result is output, wherein the policy network is pre-trained based on the sample evaluation value, and the sample evaluation value is obtained based on the sample initial decision result obtained after making a behavioral decision on the sample state information.

[0033] According to an embodiment of the present disclosure, the behavior decision method further includes:

[0034] In the case where the second image frame collected at any time t′ before the tth time is a sample image with fog, defogging the sample image with fog to obtain a sample defogging image;

[0035] Based on the attention mechanism, the state of the sample dehazed image is detected, and the sample state information is output, wherein the sample state information is used to represent the environment information of the target object in the second image frame;

[0036] Based on the initial policy network of deep reinforcement learning, behavioral decisions are made on the sample state information to obtain the sample initial decision results, where the sample initial decision results are used to represent the action information that the target object needs to perform from the t′-1 moment to the t′ moment;

[0037] Using the initial prediction network of deep reinforcement learning, the initial decision result of the sample is predicted to obtain the predicted initial state information, where the predicted initial state information is used to represent the environmental information of the target object predicted at the t′th moment;

[0038] Based on the predicted initial state information and the label matching the predicted initial state information, an initial prediction network is trained to obtain a trained prediction network;

[0039] Determining a sample evaluation value for an initial decision result of the sample based on the prediction state information obtained from the trained prediction network; and

[0040] Based on the sample evaluation value and the policy gradient, the parameters of the initial policy network are updated until the second preset condition is met to obtain the policy network.

[0041] A second aspect of the present disclosure provides a behavior decision-making device, comprising:

[0042] A defogging module, for performing defogging processing on the foggy image to obtain a defogged image when the first image frame collected at time t is a foggy image;

[0043] A detection module is used to perform state detection on the defogging image based on an attention mechanism and output state information, wherein the state information is used to represent the environment information of the target object at the tth moment;

[0044] A decision module, which is used to make behavioral decisions on state information based on deep reinforcement learning to obtain an initial decision result, wherein the initial decision result is used to represent the action information that the target object needs to perform from the tth moment to the t+1th moment; and

[0045] The determination module is used to determine the initial decision result as the target decision result when it is determined that the initial decision result meets the first preset condition.

[0046] A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above-mentioned behavior decision-making method.

[0047] The fourth aspect of the present disclosure further provides a computer-readable storage medium having a computer program or instructions stored thereon, which implements the steps of the above-mentioned behavior decision-making method when the above-mentioned computer program or instructions are executed by a processor.

[0048] The fifth aspect of the present disclosure also provides a computer program product, including a computer program or instructions, which implement the steps of the above-mentioned behavior decision-making method when executed by a processor.

[0049] The sixth aspect of the present disclosure also provides an autonomous driving vehicle, comprising the above-mentioned electronic device.

[0050] According to the embodiments of the present disclosure, for scenes with fog, defogging and state detection are combined with deep reinforcement learning, so that defogging and state detection are used as scene recognition, which can not only achieve the effect of defogging, but also realize the effect of identifying the environment in which the target object is located based on the attention mechanism. For the identified real-time scene, deep reinforcement learning is used for real-time decision-making, and the initial decision result can be obtained in time to achieve real-time decision-making. In addition, only when the first preset condition is met, the initial decision result is determined as the target decision result, which can enable the target object to maintain safe and stable operation in foggy weather. At least partially solves the problem that the existing defogging method based on prior and learning mapping has large errors, and when making decisions on the driving behavior of the target object, the error is transmitted, affecting the safe operation of the target object. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0052] Figure 1 A diagram schematically illustrates an application scenario of a behavior decision method, apparatus, device, medium, program product, and autonomous driving vehicle according to an embodiment of the present disclosure;

[0053] Figure 2 A flowchart of a behavior decision-making method according to an embodiment of the present disclosure is schematically shown;

[0054] Figure 3 A schematic diagram of the structure of a behavior decision model according to another embodiment of the present disclosure is shown;

[0055] Figure 4 A flowchart schematically shows a method for training a behavior decision model according to an embodiment of the present disclosure;

[0056] Figure 5 A structural block diagram of a behavior decision-making device according to an embodiment of the present disclosure is schematically shown; and

[0057] Figure 6 A block diagram of an electronic device suitable for implementing a behavior decision-making method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0058] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0059] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0060] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0061] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0062] In the technical solution of the present disclosure, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0063] In the scenario of using personal information for automated decision-making, the methods, devices, and systems provided by the embodiments of the present disclosure provide users with corresponding operation portals for users to choose to agree or reject the automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered. The expression "automated decision-making" here refers to the activity of automatically analyzing and evaluating a person's behavioral habits, interests and hobbies, or economic, health, credit status, etc. through computer programs, and making decisions. The expression "expert decision-making" here refers to the activity of making decisions by people who specialize in a certain field, have specialized experience, knowledge and skills, and have reached a certain level of professionalism.

[0064] In the technical solution of the embodiment of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0065] In the process of implementing the present disclosure, it is found that foggy weather is the most challenging in vision-based autonomous driving solutions. For autonomous driving, in foggy weather, it is difficult for the camera to identify lane lines and other traffic participants.

[0066] In order to solve the decision-making problem of autonomous driving in foggy scenes, related technologies are based on prior and learning mapping methods. However, these methods have large defogging errors due to the lack of cognitive awareness of scene depth and the inability to capture length dependence. It is difficult to obtain good results when dealing with such foggy scenes in decision-making algorithms that rely on rules. For example, the prior-based method is based on certain assumptions and estimates, which leads to large defogging errors. If the output of the prior-based method is used as the input of the deep reinforcement learning algorithm, it will bring greater errors to the decision-making algorithm and affect the driving safety of autonomous vehicles. The learning mapping-based method usually learns a mapping. The data set is usually given a large number of foggy images and their corresponding images after defogging, which will be overly dependent on the training process and the quality of the data set. Whether it is a prior-based method or a learning mapping-based method, when the results are transmitted to the rule-based decision algorithm, the errors will be transmitted, that is, the downstream decision results are greatly affected by the upstream errors.

[0067] The embodiments of the present disclosure provide a behavior decision method, device, equipment, medium, program product and autonomous driving vehicle. The method includes: when the first image frame collected at the tth time is a foggy image, defogging the foggy image to obtain a defogging image; based on the attention mechanism, performing state detection on the defogging image to obtain state information, wherein the state information is used to characterize the environment information of the target object at the tth time; based on deep reinforcement learning, performing behavior decision on the state information to obtain an initial decision result, wherein the initial decision result is used to characterize the action information that the target object needs to perform from the tth time to the t+1th time; and when it is determined that the initial decision result meets the first preset condition, determining the initial decision result as the target decision result.

[0068] Figure 1 The application scenario diagram of the behavior decision method, apparatus, device, medium, program product and autonomous driving vehicle according to the embodiments of the present disclosure is schematically shown.

[0069] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a first target object 101, a second target object 102, a third target object 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first target object 101, the second target object 102, the third target object 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0070] The user can use at least one of the first target object 101, the second target object 102, and the third target object 103 to interact with the server 105 through the network 104 to receive or send messages, etc. The first target object 101, the second target object 102, and the third target object 103 may be equipped with a controller for controlling the travel of the target object, a sensor for sensing the surrounding environment, a high-precision map navigation and positioning for guiding the travel of the target object, and a collection device for capturing visual images in front of the travel direction of the target object. The collection device may include but is not limited to a smart camera, a smart camera, etc.

[0071] The first target object 101, the second target object 102, and the third target object 103 can be autonomous vehicles or unmanned vehicles, or wheeled mobile robots, etc., with high-precision map navigation and positioning, sensors, controllers, smart cameras or smart cameras.

[0072] The server 105 may be a server that provides various services, such as a background management server (only as an example) that can provide support for navigation or automatic driving for the first target object 101, the second target object 102, and the third target object 103. The background management server may analyze the image with fog based on the first image frame collected at the tth moment, and feed back the processing result (such as the target decision result, etc.) to the target object.

[0073] It should be noted that the behavior decision method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the behavior decision device provided in the embodiment of the present disclosure can generally be set in the server 105. The behavior decision method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first target object 101, the second target object 102, the third target object 103 and / or the server 105. Accordingly, the behavior decision device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the first target object 101, the second target object 102, the third target object 103 and / or the server 105.

[0074] It should be understood that Figure 1 The number of target objects, networks and servers in the embodiment is only for illustration. Any number of target objects, networks and servers may be provided as required.

[0075] The following will be based on Figure 1 The scene described by Figure 2~Figure 4 The behavior decision-making method of the disclosed embodiment is described in detail.

[0076] Figure 2 The flowchart of the behavior decision-making method according to the embodiment of the present disclosure is schematically shown.

[0077] like Figure 2 As shown, the behavior decision method 200 of this embodiment includes operations S210 to S240.

[0078] In operation S210, when the first image frame acquired at time t is a foggy image, defogging is performed on the foggy image to obtain a defogged image.

[0079] In operation S220, based on the attention mechanism, state detection is performed on the dehazed image to obtain state information.

[0080] In operation S230, based on deep reinforcement learning, a behavior decision is made on the state information to obtain an initial decision result.

[0081] In operation S240, in a case where it is determined that the initial decision result satisfies the first preset condition, the initial decision result is determined as a target decision result.

[0082] According to an embodiment of the present disclosure, when the target object is moving, an acquisition device installed on the target object, such as a smart camera, can be used to acquire image frames. The target object may include but is not limited to a vehicle capable of autonomous driving or an unmanned vehicle, or a wheeled mobile robot, etc.

[0083] According to an embodiment of the present disclosure, a defogging image can be used to characterize an image with higher definition than an image with fog. Feature extraction can be performed on the image with fog to obtain image illumination information, image fog information, and a defogging image. The defogging image can be reconstructed based on the image illumination information, image fog information, and the defogging image.

[0084] According to an embodiment of the present disclosure, the state information may be used to characterize the environment information of the target object at the tth moment. For example, the environment information of the target object may be, but is not limited to, obstacle information in front of the target object's traveling direction, lane information indicating the lane in which the target object is located, and information indicating whether the target object is passable. The passable information may include traffic signal information.

[0085] In a specific example, the state information can be represented by state, state=(obstacles, J, Traffic), where obstacles represent detected obstacles, J represents detected lanes, and Traffic represents detected traffic light information.

[0086] According to an embodiment of the present disclosure, the self-attention mechanism can be integrated into a neural network architecture (Transformer) for natural language processing, and then enabled to detect targets in the dehazed image. The target objects may include, but are not limited to, obstacles, lanes, traffic lights, etc.

[0087] For example, the encoder and decoder in Transformer can be composed of a self-attention layer and a feed-forward network. The dehazed image can be input into the encoder for encoding, and then decoded by the decoder to obtain a feature vector, and the feature vector is passed through a layer of activation function to obtain state information.

[0088] According to an embodiment of the present disclosure, the initial decision result is used to characterize the action information that the target object needs to perform from time t to time t+1. For example, the action information may include the moving speed information of the target object or the moving acceleration information of the target object.

[0089] According to the embodiments of the present disclosure, deep learning perception state information can be utilized in deep reinforcement learning, and reinforcement learning in deep reinforcement learning can be utilized to make behavioral decisions on the perceived state information to obtain an initial decision result.

[0090] According to an embodiment of the present disclosure, the first preset condition may include a condition that enables the target object to maintain a safe state when executing the action information. For example, when the target object moves according to the initial decision result, the target object does not encounter abnormal conditions such as collision and / or bumping, then the initial decision result can be considered to meet the first preset condition. When the target object moves according to the initial decision result, the target object encounters abnormal conditions such as collision and / or bumping, then the initial decision result can be considered to not meet the first preset condition.

[0091] According to the embodiments of the present disclosure, the initial decision result can be subjected to a safety assessment, and if the assessment shows that the target object can maintain a safe state by moving based on the initial decision result, the initial decision result is used as the target decision result. Otherwise, a new decision needs to be made.

[0092] According to the embodiments of the present disclosure, for scenes with fog, defogging and state detection are combined with deep reinforcement learning, so that defogging and state detection are used as scene recognition, which can not only achieve the effect of defogging, but also realize the effect of identifying the environment in which the target object is located based on the attention mechanism. For the identified real-time scene, deep reinforcement learning is used for real-time decision-making, and the initial decision result can be obtained in time to achieve real-time decision-making. In addition, only when the first preset condition is met, the initial decision result is determined as the target decision result, which can enable the target object to maintain safe and stable operation in foggy weather. At least partially solves the problem that the existing defogging method based on prior and learning mapping has large errors, and when making decisions on the driving behavior of the target object, the error is transmitted, affecting the safe operation of the target object.

[0093] According to the embodiments of the present disclosure, for the above Figure 2 The operation S210 shown, when the first image frame collected at the tth time is an image with fog, performs defogging processing on the image with fog to obtain a defogged image, which may include the operations of: extracting features from the image with fog to obtain a feature image. Performing residual prediction on the feature image to obtain a predicted residual image. Fusion of the predicted residual image and the feature image to obtain a defogged image.

[0094] According to an embodiment of the present disclosure, the tth moment may correspond to the moment when the acquisition device acquires an image when the target object is moving. When the light information of the acquired first image frame does not satisfy the preset light information and / or the clarity information does not satisfy the preset clarity information, it may be determined that the first image frame is an image with fog. The preset light information and / or preset clarity information may be determined based on the light or clarity of the image acquired in an actual foggy scene.

[0095] According to an embodiment of the present disclosure, a foggy image may be input into a deep convolutional neural network for feature extraction to obtain a feature image.

[0096] According to the embodiments of the present disclosure, a residual network can be constructed using skip connections and residual blocks, a feature image is input into the residual network, and after residual prediction, the predicted residual image and the feature image are fused based on the skip connection to obtain a defogged image. The defogged image can have higher clarity than the foggy image.

[0097] According to the embodiments of the present disclosure, when the target object is traveling in the direction of travel, it is necessary to plan the driving path. Although the path can be planned in advance, there may be sudden scenes during driving. If the path can be planned in real time, the operation safety can be improved. In foggy scenes, the collected image may cause deviations in the planned path due to unclearness, resulting in operation failures. Therefore, by defogging the identified foggy image, the defogging effect can be achieved.

[0098] According to the embodiments of the present disclosure, for the above Figure 2 The operation S230 shown, based on deep reinforcement learning, makes a behavior decision on the state information to obtain an initial decision result, which may include the operation of: inputting the state information into a policy network in deep reinforcement learning, and outputting the initial decision result.

[0099] According to an embodiment of the present disclosure, the policy network is pre-trained based on sample evaluation values, and the sample evaluation values ​​are obtained based on sample initial decision results obtained after behavioral decisions are made on sample state information.

[0100] According to an embodiment of the present disclosure, the sample evaluation value can be used to evaluate the decision result generated by the policy network. The sample state information can be obtained by performing state detection on the sample defogging image based on the attention mechanism. The sample defogging image can be obtained by defogging the sample image with fog.

[0101] According to the embodiments of the present disclosure, by training the policy network based on the sample evaluation values, the policy network can make decisions for the action information that needs to be performed by the target object by training its own decision-making ability, rather than relying solely on the defogging effect of the sample defogging image, thereby avoiding safety operation problems caused by large defogging errors.

[0102] According to an embodiment of the present disclosure, the behavior decision method may include the following: Figure 2 In addition to the operations S210 to S240 shown, the operation may also include: when the second image frame collected at any time t′ before the tth time is a sample image with fog, the sample image with fog is defogged to obtain a sample defogged image. Based on the attention mechanism, the state of the sample defogged image is detected, and the sample state information is output, wherein the sample state information is used to characterize the environmental information of the target object in the second image frame. Based on the initial policy network of deep reinforcement learning, a behavior decision is made on the sample state information to obtain the sample initial decision result. Using the initial prediction network of deep reinforcement learning, the sample initial decision result is predicted to obtain the predicted initial state information. Based on the predicted initial state information and the label matching the predicted initial state information, the initial prediction network is trained to obtain a trained prediction network. Based on the predicted state information obtained by the trained prediction network, the sample evaluation value for the sample initial decision result is determined. Based on the sample evaluation value and the policy gradient, the parameters of the initial policy network are updated until the second preset condition is met to obtain the policy network.

[0103] According to an embodiment of the present disclosure, the sample initial decision result is used to characterize the action information that the target object needs to perform from the t′-1th moment to the t′th moment.

[0104] According to an embodiment of the present disclosure, the predicted initial state information is used to characterize the predicted environment information of the target object at the time t′.

[0105] According to an embodiment of the present disclosure, time t′ may be any time occurring before time t.

[0106] According to the embodiments of the present disclosure, feature extraction can be performed on a sample image with fog to obtain a sample feature image. Residual prediction can be performed on the sample feature image to obtain a sample prediction residual image. The sample prediction residual image and the sample feature image can be fused to obtain a sample defogging image.

[0107] According to an embodiment of the present disclosure, the sample dehazed image can be input into a Transformer that incorporates a self-attention mechanism, and the environmental information of the target in the sample dehazed image can be detected to obtain sample state information.

[0108] According to an embodiment of the present disclosure, an initial policy network and an initial prediction network of deep reinforcement learning may be defined. The initial policy network is used to generate a decision result, and the initial prediction network is used to predict the decision result generated by the initial policy network.

[0109] According to an embodiment of the present disclosure, a preset amount of sample data can be obtained by using a sample image with fog collected at any time t' before the tth time, and using an initial strategy network to interact between the environment and the real environment. The sample data may include sample state information at any time before the tth time, sample initial decision results, sample estimation values, and sample state information at time t'-1 before any time t' before the tth time. The sample data may be stored in an experience playback buffer.

[0110] According to an embodiment of the present disclosure, sample data can be randomly sampled from the experience replay buffer, and the initial decision results of the samples in the sampled sample data can be predicted using the initial prediction network of deep reinforcement learning to obtain the predicted initial state information. The sample state information at time t′-1 before any time t′ before time t in the sampled sample data can be used as a label that matches the predicted initial state information. The positioning error (TD error) can be determined based on the sample state information at time t′-1 before any time t′ before time t in the sampled sample data and the predicted initial state information. By minimizing the positioning error, the sample state information at time t′-1 before any time t′ before time t is approached. The network parameters of the initial prediction network can be updated using optimization algorithms such as stochastic gradient descent until the determined positioning error value satisfies the preset error value, and a trained prediction network is obtained.

[0111] According to an embodiment of the present disclosure, the sample initial decision result can be input into a trained prediction network to obtain prediction state information. Based on the prediction state information and the sample initial decision result, the target object can be simulated to obtain a sample simulation result. The sample simulation result is used to characterize the sample driving information when the target object executes the sample initial decision result. Based on the sample simulation result and the preset evaluation function, a sample evaluation value for the sample initial decision result is determined.

[0112] According to an embodiment of the present disclosure, the policy gradient can be used to characterize the weight value of the objective function of the policy network, and the objective function of the policy network can use the objective function of the existing policy network, which is not specifically limited here. The second preset condition may include that the number of updates reaches a predetermined number of training steps or the sample evaluation value reaches a threshold. The parameters of the policy network can be continuously updated according to the sample evaluation value and the weight value of the objective function until the number of updates reaches a predetermined number of training steps or until the sample evaluation value reaches a threshold.

[0113] According to the embodiments of the present disclosure, the present application can improve the utilization of experience data obtained at any time t′ before the tth time by constructing an additional prediction network for the deep reinforcement learning network, and updating the parameters of the policy network based on the trained prediction network, so that the deep reinforcement learning network can not only focus on the generation of strategies but also on the evaluation of the generated strategies.

[0114] According to an embodiment of the present disclosure, the defogging image may include an image of a lane in front of the traveling direction of the target object.

[0115] Behavioral decision-making methods can include Figure 2 In addition to the operations S210 to S240 shown, the operation may also include: determining the length of the lane in the lane image based on the lane image in front of the target object in the direction of travel. Based on the initial decision result and the length of the lane, simulating the target object to obtain a simulation result. Based on the simulation result, determining whether the initial decision result meets the first preset condition.

[0116] According to an embodiment of the present disclosure, the simulation result can be used to characterize the driving information of the target object when executing the initial decision result. The driving information can be used to characterize the driving condition of the target object.

[0117] According to an embodiment of the present disclosure, based on the driving information when the target object executes the initial decision result, it can be determined whether the target object has an abnormal situation such as a collision and / or bumping during driving. When it is determined that the target object has an abnormal situation such as a collision and / or bumping during driving, the first preset condition is not satisfied. When it is determined that the target object has not an abnormal situation such as a collision and / or bumping during driving, the first preset condition is satisfied.

[0118] According to an embodiment of the present disclosure, determining whether the initial decision result satisfies the first preset condition based on the simulation result may include the following operations: determining an evaluation value for the initial decision result based on the simulation result and a preset evaluation function. If it is determined that the evaluation value exceeds a threshold, determining that the initial decision result satisfies the first preset condition. If it is determined that the evaluation value does not exceed the threshold, determining that the initial decision result does not satisfy the first preset condition.

[0119] According to an embodiment of the present disclosure, the preset evaluation function may be a penalty function for collision and / or bumping.

[0120] According to the embodiments of the present disclosure, by simulating the target object, the safe and stable operation of the target object can be further determined through the simulation results.

[0121] According to an embodiment of the present disclosure, the initial decision result may include lateral acceleration information and longitudinal acceleration information. The simulation result may include driving speed information and driving collision situation information when the initial decision result is executed.

[0122] According to an embodiment of the present disclosure, based on the initial decision result and the length of the lane, simulating the target object to obtain the simulation result may include the following operations: generating a simulation execution instruction according to the lateral acceleration information, the longitudinal acceleration information and the speed information of the target object at the tth moment; and collecting statistics on the driving speed information and driving collision situation information of the target object during the process of executing the simulation execution instruction.

[0123] According to an embodiment of the present disclosure, the simulation execution instruction is used to indicate the length of the lane in which the target object is traveling.

[0124] According to the embodiments of the present disclosure, by statistically analyzing the driving speed information and driving collision situation information of the target object during the execution of the simulated execution instruction, a quantitative evaluation of the initial decision result takes into account both the driving safety and driving stability of the target object, thereby improving the evaluation accuracy.

[0125] According to an embodiment of the present disclosure, the preset evaluation function may include a safety constraint item, a speed evaluation item, a comfort evaluation item, and a detection distance evaluation item.

[0126] According to an embodiment of the present disclosure, determining an evaluation value for an initial decision result based on simulation results and a preset evaluation function may include the following operations: determining the value of a safety constraint item based on driving collision situation information and preset safety constraints. Determining the driving speed information when the target object completes executing the simulation execution instruction as the value of the speed evaluation item. Determining the value of a comfort evaluation item based on the driving speed information corresponding to a preset time interval during the process of the target object executing the simulation execution instruction. Determining the length of the lane as the value of the detection distance evaluation item. Summing the values ​​of the safety constraint item, the speed evaluation item, the comfort evaluation item, and the detection distance evaluation item to obtain an evaluation value for the initial decision result.

[0127] According to an embodiment of the present disclosure, the preset safety constraint condition may include a penalty constraint for the initial decision result when it is determined that the target object has collided, and an incentive constraint for the initial decision result when it is determined that the target object has not collided.

[0128] According to an embodiment of the present disclosure, the preset time interval may be an empirical value. The difference between the speed of the target object at any moment in the process of executing the simulation execution instruction and the speed at the moment of the preset time interval from any moment may be determined, and the value of the comfort evaluation item may be obtained according to the negative number of the absolute value of the difference.

[0129] For example, the preset evaluation function may be as shown in the following formula (1):

[0130] ; (1)

[0131] in, , It is used to represent the safety constraint item. When it is determined that the target object collides, the value of the safety constraint item is -10000. When it is determined that the target object does not collide, the value of the safety constraint item is 10. , Used to represent the speed evaluation item. The value of the speed evaluation item is The running speed of the moment , No. The time represents the time when the target object completes the simulation execution instruction. , represents the comfort evaluation item, Indicates The running speed at the moment.

[0132] ,

[0133] Used to represent the detection distance evaluation item, Indicates the length of the lane.

[0134] According to the embodiments of the present disclosure, the quantitative evaluation of the initial decision result, in addition to considering the driving safety and driving stability of the target object, also takes into account the speed and the length of the lane in the defogging image, thereby improving the quantitative evaluation system from multiple aspects and thereby improving the evaluation accuracy.

[0135] Figure 3 The structural diagram of a behavior decision model according to another embodiment of the present disclosure is schematically shown.

[0136] like Figure 3 As shown, we can use Figure 3 The behavior decision model shown implements the behavior decision method. The behavior decision model may include a deep convolutional neural network (CNN), a residual network (Residual Network), an attention network (Transformer), and an improved approximate policy optimization algorithm Q-PPO. Q-PPO includes a policy network of the approximate policy optimization algorithm (PPO) and a constructed prediction network.

[0137] like Figure 3As shown, when the first image frame collected at the tth moment is an image 301 with fog, the image 301 with fog is input into CNN, and a feature image 302 is output. The feature image 302 is input into ResidualNetwork, and a defogging image 303 is output. The defogging image 303 is input into Transformer, and state information 304 is output. The state information 304 is input into Q-PPO, and an initial decision result 305 is output. The target object is simulated based on the initial decision result 305 to obtain a simulation result, and an evaluation value 306 for the initial decision result 305 is determined based on the simulation result and a preset evaluation function. The parameters of the policy network of Q-PPO can be updated in real time based on the evaluation value 306 for the initial decision result 305.

[0138] According to the embodiments of the present disclosure, Figure 3 The behavior decision model shown is trained using the training method of the behavior decision model shown in FIG4 .

[0139] Figure 4 The flowchart of the method for training the behavior decision model according to the embodiment of the present disclosure is schematically shown.

[0140] like Figure 4 As shown, the behavior decision model training method 400 of this embodiment may include operations S401 to S406.

[0141] In operation S401, when the second image frame collected at any time t′ before the tth time is a sample image with fog, the sample image with fog is sequentially input into the initial deep convolutional neural network, the initial residual network, and the initial attention network, and the sample state information is output.

[0142] In operation S402, the sample state information is input into the initial policy network, and the sample initial decision result is output.

[0143] In operation S403, the sample initial decision result is input into the initial prediction network, and the prediction initial state information is output.

[0144] In operation S404, an initial prediction network is trained based on the predicted initial state information and a label matching the predicted initial state information to obtain a trained prediction network.

[0145] In operation S405, a sample evaluation value for an initial decision result of the sample is determined based on the prediction state information obtained by the trained prediction network.

[0146] In operation S406, based on the sample evaluation values, the parameters of the initial deep convolutional neural network, the initial residual network, the initial attention network, and the initial policy network are updated to obtain a behavior decision model.

[0147] According to the embodiments of the present disclosure, by adjusting the parameters of the initial deep convolutional neural network, the initial residual network, the initial attention network and the initial policy network at the same time, the training speed of the behavior decision model can be accelerated. The policy network can make a better decision even if the clarity of the sample defogging image output after the sample image passes through the deep convolutional neural network, the residual network and the attention network in sequence is not high. The requirements for defogging processing are reduced.

[0148] According to another embodiment of the present disclosure, after executing operation S405, only the parameters of the initial policy network may be updated based on the sample evaluation value, and the behavior decision model may be obtained as long as the decision result generated by the policy network satisfies the first preset condition.

[0149] Based on the above behavior decision method, the present disclosure also provides a behavior decision device. Figure 5 The device is described in detail.

[0150] Figure 5 The structural block diagram of the behavior decision-making device according to an embodiment of the present disclosure is schematically shown.

[0151] like Figure 5 As shown, the behavior decision device 500 of this embodiment includes a defogging module 510 , a detection module 520 , a decision module 530 and a determination module 540 .

[0152] The defogging module 510 is used to perform defogging on the image with fog to obtain a defogged image when the first image frame collected at time t is a foggy image. In one embodiment, the defogging module 510 can be used to perform the operation S210 described above, which will not be repeated here.

[0153] The detection module 520 is used to perform state detection on the defogging image based on the attention mechanism and output state information, wherein the state information is used to characterize the environment information of the target object at the tth moment. In one embodiment, the detection module 520 can be used to perform the operation S220 described above, which will not be repeated here.

[0154] The decision module 530 is used to make a behavior decision on the state information based on deep reinforcement learning to obtain an initial decision result, wherein the initial decision result is used to represent the action information that the target object needs to perform from the tth moment to the t+1th moment. In one embodiment, the decision module 530 can be used to perform the operation S230 described above, which will not be repeated here.

[0155] The determination module 540 is used to determine the initial decision result as the target decision result when it is determined that the initial decision result meets the first preset condition. In one embodiment, the determination module 540 can be used to perform the operation S240 described above, which will not be described in detail here.

[0156] According to an embodiment of the present disclosure, the defogging image includes an image of a lane in front of the target object in the direction of travel;

[0157] The behavior decision device 500 may further include a length determination module, a simulation module, an evaluation value determination module and a result determination module.

[0158] The length determination module is used to determine the length of the lane in the lane image based on the lane image in front of the target object in the traveling direction.

[0159] The simulation module is used to simulate the target object based on the initial decision result and the length of the lane to obtain a simulation result, wherein the simulation result is used to characterize the driving information when the target object executes the initial decision result.

[0160] The evaluation value determination module is used to determine the evaluation value for the initial decision result based on the simulation result and the preset evaluation function.

[0161] The result determination module is used to determine that the initial decision result meets the first preset condition when it is determined that the evaluation value exceeds the threshold.

[0162] According to an embodiment of the present disclosure, the initial decision result includes lateral acceleration information and longitudinal acceleration information, and the simulation result includes driving speed information and driving collision situation information when the initial decision result is executed;

[0163] The simulation module may include an instruction generation unit and a statistics unit.

[0164] The instruction generation unit is used to generate a simulation execution instruction according to the lateral acceleration information, the longitudinal acceleration information and the speed information of the target object at the tth moment, wherein the simulation execution instruction is used to indicate the length of the lane where the target object travels.

[0165] The statistical unit is used to collect statistics on the driving speed information and driving collision situation information of the target object during the process of executing the simulation execution instruction.

[0166] According to an embodiment of the present disclosure, the preset evaluation function includes a safety constraint item, a speed evaluation item, a comfort evaluation item, and a detection distance evaluation item;

[0167] The evaluation value determination module may include a first determination subunit, a second determination subunit, a third determination subunit, a fourth determination subunit, and a fifth determination subunit.

[0168] The first determining subunit is used to determine the value of the safety constraint item according to the driving collision situation information and the preset safety constraint condition.

[0169] The second determining subunit is used to determine the driving speed information of the target object when the target object completes executing the simulation execution instruction as the value of the speed evaluation item.

[0170] The third determining subunit determines the value of the comfort evaluation item according to the driving speed information corresponding to the preset time interval during the process of the target object executing the simulation execution instruction.

[0171] The fourth determining subunit determines the length of the lane as a value of the detection distance evaluation item.

[0172] The fifth determination subunit sums the value of the safety constraint item, the value of the speed evaluation item, the value of the comfort evaluation item, and the value of the detection distance evaluation item to obtain an evaluation value for the initial decision result.

[0173] According to an embodiment of the present disclosure, the defogging module 510 may include: an extraction unit, a prediction unit, and a fusion unit.

[0174] The extraction unit is used to extract features from the image with fog to obtain a feature image.

[0175] The prediction unit is used to perform residual prediction on the feature image to obtain a predicted residual image.

[0176] The fusion unit fuses the predicted residual image and the feature image to obtain a dehazed image.

[0177] According to an embodiment of the present disclosure, the decision module 530 may include a decision subunit.

[0178] The decision subunit is used to input state information into the policy network in deep reinforcement learning and output an initial decision result, wherein the policy network is pre-trained based on the sample evaluation value, and the sample evaluation value is obtained based on the sample initial decision result obtained after making a behavioral decision on the sample state information.

[0179] According to an embodiment of the present disclosure, the behavior decision device 500 may further include: a sample defogging module, a sample detection module, a sample decision module, a prediction module, a training module, a sample determination module and an update module.

[0180] The sample defogging module is used to perform defogging processing on the sample image with fog to obtain a sample defogging image when the second image frame collected at any time t′ before the tth time is a sample image with fog.

[0181] The sample detection module is used to perform state detection on the sample dehazed image based on the attention mechanism, and output sample state information, wherein the sample state information is used to characterize the environment information of the target object in the second image frame.

[0182] The sample decision module is used to make behavioral decisions on the sample state information based on the initial policy network of deep reinforcement learning to obtain the sample initial decision result, where the sample initial decision result is used to characterize the action information that the target object needs to perform from the t′-1 moment to the t′ moment.

[0183] The prediction module is used to use the initial prediction network of deep reinforcement learning to predict the initial decision results of the sample and obtain the predicted initial state information, wherein the predicted initial state information is used to characterize the environmental information of the target object predicted at the t′th moment.

[0184] The training module is used to train the initial prediction network based on the predicted initial state information and the label matching the predicted initial state information to obtain a trained prediction network.

[0185] The sample determination module is used to determine the sample evaluation value for the sample initial decision result based on the prediction state information obtained by the trained prediction network.

[0186] The update module updates the parameters of the initial policy network based on the sample evaluation value and the policy gradient until the first preset condition is met to obtain the policy network.

[0187] According to an embodiment of the present disclosure, any multiple modules of the defogging module 510, the detection module 520, the decision module 530 and the determination module 540 can be combined in one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the defogging module 510, the detection module 520, the decision module 530 and the determination module 540 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in an appropriate combination of any of them. Alternatively, at least one of the defogging module 510 , the detection module 520 , the decision module 530 , and the determination module 540 may be at least partially implemented as a computer program module, and when the computer program module is executed, a corresponding function may be performed.

[0188] Figure 6 A block diagram of an electronic device suitable for implementing a behavior decision-making method according to an embodiment of the present disclosure is schematically shown.

[0189] like Figure 6 As shown, the electronic device 600 according to an embodiment of the present disclosure includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 to a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a dedicated microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include an onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0190] In RAM 603, various programs and data required for the operation of electronic device 600 are stored. Processor 601, ROM 602 and RAM 603 are connected to each other via bus 604. Processor 601 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 602 and / or RAM 603. It should be noted that the program can also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in one or more memories.

[0191] According to an embodiment of the present disclosure, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to the bus 604. The electronic device 600 may further include one or more of the following components connected to the I / O interface 605: an input portion 606 including a keyboard, a mouse, etc.; an output portion 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 608 including a hard disk, etc.; and a communication portion 609 including a network interface card such as a LAN card, a modem, etc. The communication portion 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage portion 608 as needed.

[0192] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0193] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the ROM 602 and / or RAM 603 described above and / or one or more memories other than ROM 602 and RAM 603.

[0194] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the method provided by the embodiment of the present disclosure.

[0195] The above functions defined in the system / device of the embodiment of the present disclosure are performed when the computer program is executed by the processor 601. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0196] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 609, and / or installed from a removable medium 611. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0197] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the processor 601, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, apparatus, module, unit, etc. described above can be implemented by a computer program module.

[0198] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0199] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0200] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations and / or combinations are not explicitly described in the present disclosure. In particular, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways without departing from the spirit and teachings of the present disclosure. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0201] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. The scope of the present disclosure is defined by the attached claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A behavioral decision-making method, characterized in that: The method comprises: When the first image frame collected at the time t is a foggy image, defogging the foggy image to obtain a defogged image; Based on the attention mechanism, the state of the defogging image is detected to obtain state information, wherein the state information is used to represent the environment information of the target object at the t-th moment; Based on deep reinforcement learning, a behavior decision is made on the state information to obtain an initial decision result, wherein the initial decision result is used to represent the action information that the target object needs to perform from the tth moment to the t+1th moment; and When it is determined that the initial decision result satisfies the first preset condition, the initial decision result is determined as the target decision result.

2. The method according to claim 1, characterized in that The defogging image includes an image of a lane in front of the target object in a direction of travel; The method further comprises: Determining the length of the lane in the lane image based on the lane image in front of the target object in the direction of travel; Based on the initial decision result and the length of the lane, simulating the target object to obtain a simulation result, wherein the simulation result is used to characterize the driving information of the target object when executing the initial decision result; Based on the simulation result and a preset evaluation function, determining an evaluation value for the initial decision result; and In the case where it is determined that the evaluation value exceeds a threshold, it is determined that the initial decision result meets the first preset condition.

3. The method according to claim 2, characterized in that The initial decision result includes lateral acceleration information and longitudinal acceleration information, and the simulation result includes driving speed information and driving collision situation information when executing the initial decision result; The simulating the target object based on the initial decision result and the length of the lane to obtain a simulation result includes: generating a simulation execution instruction according to the lateral acceleration information, the longitudinal acceleration information and the speed information of the target object at the t-th time, wherein the simulation execution instruction is used to indicate the length of the lane traveled by the target object; and The driving speed information and the driving collision situation information of the target object during the process of executing the simulation execution instruction are counted.

4. The method according to claim 3, characterized in that The preset evaluation function includes a safety constraint item, a speed evaluation item, a comfort evaluation item and a detection distance evaluation item; The step of determining an evaluation value for the initial decision result based on the simulation result and a preset evaluation function includes: Determining the value of the safety constraint item according to the driving collision situation information and the preset safety constraint condition; Determining the driving speed information when the target object completes executing the simulation execution instruction as the value of the speed evaluation item; Determining the value of the comfort evaluation item according to the driving speed information corresponding to the preset time interval during the process of the target object executing the simulation execution instruction; determining the length of the lane as the value of the detection distance evaluation item; and The value of the safety constraint item, the value of the speed evaluation item, the value of the comfort evaluation item, and the value of the detection distance evaluation item are summed to obtain the evaluation value for the initial decision result.

5. The method according to any one of claims 1 to 4, characterized in that: When the first image frame collected at time t is a foggy image, performing defogging processing on the foggy image to obtain a defogged image includes: Performing feature extraction on the foggy image to obtain a feature image; Performing residual prediction on the feature image to obtain a predicted residual image; and The predicted residual image and the feature image are fused to obtain the defogging image.

6. The method according to any one of claims 1 to 4, characterized in that: The method of performing a behavior decision on the state information based on deep reinforcement learning to obtain an initial decision result includes: The state information is input into the policy network in the deep reinforcement learning, and the initial decision result is output, wherein the policy network is pre-trained based on the sample evaluation value, and the sample evaluation value is obtained based on the sample initial decision result obtained after making a behavioral decision on the sample state information.

7. The method according to claim 6, characterized in that The method further comprises: In the case where the second image frame collected at any time t′ before the tth time is a sample image with fog, defogging the sample image with fog to obtain a sample defogging image; Based on the attention mechanism, a state detection is performed on the sample defogging image, and sample state information is output, wherein the sample state information is used to characterize the environment information where the target object in the second image frame is located; Based on the initial policy network of deep reinforcement learning, a behavior decision is made on the sample state information to obtain a sample initial decision result, wherein the sample initial decision result is used to represent the action information that the target object needs to perform from the t′-1th moment to the t′th moment; Using an initial prediction network of deep reinforcement learning, predicting the initial decision result of the sample to obtain predicted initial state information, wherein the predicted initial state information is used to represent the environmental information of the target object predicted at the time t′; Based on the predicted initial state information and the label matching the predicted initial state information, training the initial prediction network to obtain a trained prediction network; Determining a sample evaluation value for the sample initial decision result based on the prediction state information obtained by the trained prediction network; and Based on the sample evaluation value and the policy gradient, the parameters of the initial policy network are updated until a second preset condition is met to obtain the policy network.

8. A behavior decision device, characterized in that: The device comprises: A defogging module, configured to, when the first image frame collected at time t is a foggy image, perform defogging processing on the foggy image to obtain a defogged image; A detection module, configured to perform state detection on the defogging image based on an attention mechanism, and output state information, wherein the state information is used to characterize the environment information where the target object is located at the t-th moment; A decision module, configured to make a behavior decision on the state information based on deep reinforcement learning to obtain an initial decision result, wherein the initial decision result is used to represent the action information that the target object needs to perform from the tth moment to the t+1th moment; and The determination module is used to determine the initial decision result as the target decision result when it is determined that the initial decision result meets the first preset condition.

9. An electronic device, comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

11. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

12. An autonomous driving vehicle comprising the electronic device described in claim 9.

Images