Brain-like scene safety assessment method combining process supervision and fast and slow thinking

Abstract

The invention relates to a brain-like scene safety assessment method combining process supervision and fast and slow thinking. The method comprises the following steps: firstly, designing a brain-like scene security cognition process and a scene security level division standard; then, constructing a supervision fine tuning data set and a reinforcement learning fine tuning data set of a process supervision normal form based on the brain-like scene security cognition process and a scene security level division standard; then, performing cold start on an open-source pre-training multi-modal large language model based on the supervision fine tuning data set to obtain a supervision fine tuning model; carrying out reinforcement learning training on the supervision fine tuning model based on the reinforcement learning fine tuning data set of the process supervision normal form; and finally, inputting a to-be-evaluated scene image into the trained supervision fine tuning model to obtain a scene security level. The inference process of the model can be aligned with the thinking process of human in the macroscopic level, the flexibility of the thinking process is maintained, and the interpretability of the result of each step is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of scene understanding, in particular to a brain-like scene safety evaluation method combining process supervision and fast and slow thinking. BACKGROUND

[0002] Safety, as a complex and advanced cognitive goal, is the main purpose of implementing scene understanding technology in many vertical fields. Common application scenarios of scene safety understanding include: autonomous driving, construction site monitoring, emergency rescue, embodied intelligence, etc. The result of safety understanding is the key basis for upper-level decision-making. Accurate and reliable scene safety understanding results can help humans make reasonable risk avoidance decisions and effectively reduce accidents, which is of great significance to human life and property safety. In order to achieve accurate and reliable scene safety understanding, it is necessary to comprehensively perceive the targets in the scene, mine related clues, and conduct deep reasoning combined with background knowledge.

[0003] Therefore, in the prior art, many works have studied scene safety understanding. First, a deep neural network model is used to supervise the mapping relationship between the scene and safety. However, due to the "black box" nature of the neural network, this result supervision method lacks a causal reasoning chain, and the result is unreliable. Second, a pre-trained visual language model is used to reason about the safety of the input scene. The model can output an interpretable reasoning process, but its thinking process is difficult to align with human cognitive processes, and it is easy to generate hallucinations. Third, a thinking chain is designed in advance, and a multi-modal large language visual model is fine-tuned to perform safety reasoning according to the agreed format, but this method restricts the generalization ability and reasoning flexibility of the model, making it difficult to improve performance.

[0004] Therefore, in the related art, there is an urgent need for a method that can solve the problems of insufficient reasoning flexibility, poor performance, and poor interpretability of scene safety understanding methods in actual implementation. SUMMARY

[0005] Therefore, it is necessary to provide a brain-like scene safety evaluation method combining process supervision and fast and slow thinking to solve the above technical problems.

[0006] In a first aspect, the present application provides a brain-like scene safety evaluation method combining process supervision and fast and slow thinking. The method comprises: designing a brain-like scene safety cognitive process and scene safety level classification standard; constructing a supervised fine-tuning data set and a reinforcement learning fine-tuning data set of the process supervision paradigm based on the brain-like scene safety cognitive process and the scene safety level classification standard; cold-starting an open-source pre-trained multi-modal large language model based on the supervised fine-tuning data set to obtain a supervised fine-tuning model; perform reinforcement learning training on the supervised fine-tuning model based on the reinforcement learning fine-tuning dataset of the process supervision paradigm; input a scene image to be evaluated into the supervised fine-tuning model after training to obtain a scene safety level.

[0007] Optionally, in an embodiment of the present application, the brain-like scene safety cognitive process includes entity perception, relationship extraction, overall understanding, and safety cognition.

[0008] Optionally, in an embodiment of the present application, the constructing of the supervised fine-tuning dataset and the reinforcement learning fine-tuning dataset of the process supervision paradigm based on the brain-like scene safety cognitive process includes: collecting image data based on a visual task, and generating corresponding image description data based on the image data; manually annotating the image description data; extracting entities and corresponding entity relationships from the image description data after manual annotation, and adding target detection labels and relationship extraction labels based on the entities and corresponding entity relationships, respectively; annotating safety level labels based on scene safety level division standards; constructing the supervised fine-tuning dataset and the reinforcement learning fine-tuning dataset of the process supervision paradigm based on the image data, the image description data, the target detection labels, the relationship extraction labels, and the safety level labels.

[0009] Optionally, in an embodiment of the present application, the constructing of the supervised fine-tuning dataset and the reinforcement learning fine-tuning dataset of the process supervision paradigm based on the image data, the image description data, the target detection labels, the relationship extraction labels, and the safety level labels includes: designing instructions of the supervised fine-tuning data and the reinforcement learning fine-tuning data of the process supervision paradigm; inputting the instructions as text in the supervised fine-tuning data and the reinforcement learning fine-tuning data of the process supervision paradigm, respectively, and inputting the image data as images in the supervised fine-tuning data and the reinforcement learning fine-tuning data of the process supervision paradigm, respectively; generating answers of the supervised fine-tuning data and the reinforcement learning fine-tuning data of the process supervision paradigm based on the image description data, the target detection labels, the relationship extraction labels, and the safety level labels, respectively.

[0010] Optionally, in an embodiment of the present application, the reinforcement learning training on the supervised fine-tuning model based on the reinforcement learning fine-tuning dataset of the process supervision paradigm includes: copying the supervised fine-tuning model, one as a reference model and one as a policy model; inputting text input and image input in the reinforcement learning fine-tuning dataset of the process supervision paradigm into the policy model together to generate an initial answer; designing a verifiable reward, calculating a reward value of each step of each initial answer based on the verifiable reward, and calculating an advantage value of each token of the result part of each step based on the reward value; calculating the KL divergence between the reference model and the policy model; calculating a loss value based on the advantage value and the KL divergence, training the policy model based on the loss value, and updating the parameters.

[0011] Optionally, in an embodiment of the present application, the designing a verifiable reward comprises: designing a verifiable reward for the target detection result based on the predicted target detection result and the real target detection label; designing a verifiable reward for the relationship extraction result based on the predicted target relationship and the real relationship extraction label; designing a verifiable reward for the overall understanding result based on the predicted image description and the real image description; designing a verifiable reward for the safety cognition result based on the predicted scene safety level and the real safety level label.

[0012] Optionally, in an embodiment of the present application, the calculating a reward value of each step of each initial answer based on the verifiable reward comprises: segmenting each initial answer by steps, corresponding to the brain-like scene safety cognition process; calculating a corresponding reward value based on the corresponding model prediction result, to obtain a reward set; normalizing each step reward in the reward set to obtain a final reward value.

[0013] In a second aspect, the present application also provides a brain-like scene safety evaluation device combining process supervision and fast and slow thinking. The device comprises: a brain-like scene safety cognition process and grade division standard design module for designing a brain-like scene safety cognition process and a scene safety grade division standard; a dataset construction module for constructing a supervised fine-tuning dataset and a reinforcement learning fine-tuning dataset of a process supervision paradigm based on the brain-like scene safety cognition process and the scene safety grade division standard; a model cold start module for cold starting an open source pre-trained multi-modal large language model based on the supervised fine-tuning dataset to obtain a supervised fine-tuning model; a model reinforcement learning training module for reinforcement learning training of the supervised fine-tuning model based on the reinforcement learning fine-tuning dataset of the process supervision paradigm. a scene safety evaluation module configured to input the scene image to be evaluated into the supervised fine-tuning model after training, to obtain a scene safety level.

[0014] In a third aspect, the present application also provides a computer device. The computer device comprises a memory and a processor, the memory stores a computer program, and the processor executes the steps of the method in each of the above embodiments.

[0015] In a fourth aspect, the present application also provides a computer readable storage medium. The computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the method in each of the above embodiments.

[0016] The above-described brain-like scene safety evaluation method combining process supervision and fast and slow thinking first designs a brain-like scene safety cognitive process and a scene safety level division standard. Then, based on the brain-like scene safety cognitive process and the scene safety level division standard, a supervised fine-tuning dataset and a reinforcement learning fine-tuning dataset of the process supervision paradigm are constructed. Then, based on the supervised fine-tuning dataset, an open-source pre-trained multi-modal large language model is cold-started to obtain a supervised fine-tuning model. Then, based on the reinforcement learning fine-tuning dataset of the process supervision paradigm, the supervised fine-tuning model is trained by reinforcement learning. Finally, the scene image to be evaluated is input into the supervised fine-tuning model after training to obtain a scene safety level. That is, inspired by the human hierarchical cognitive mechanism, the scene safety understanding task is divided into multiple key steps that progress layer by layer. The reasoning process of the model is aligned with the human thinking process at the macro level, and the model first performs deep slow thinking at each step. Through the paradigm of process supervision, the model realizes multiple slow thinking, and the supervised object is the conclusion of each slow thinking, corresponding to multiple key links in the human hierarchical cognitive process, so that the model is aligned with the human cognitive process at the macro level of the reasoning process, and the result of this step is summarized and output. In the training process, the slow thinking process is ignored, and the model fully utilizes the prior knowledge possessed by itself to reason flexibly, effectively avoiding the problems of insufficient reasoning or excessive reasoning of the model when facing complex reasoning tasks, reducing the occurrence of hallucinations, maintaining the flexibility of the thinking process, and improving the explainability of the result of each step. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 An application environment diagram of a brain-like scene safety evaluation method combining process supervision and fast and slow thinking in one embodiment; Figure 2 A flowchart of a brain-like scene safety evaluation method combining process supervision and fast and slow thinking in one embodiment; Figure 3A schematic diagram of a scenario safety awareness process in an embodiment; Figure 4 A schematic diagram of a data set construction method in an embodiment; Figure 5 A schematic diagram of a process of reinforcement learning training of a supervised fine-tuned model based on a fine-tuning data set of reinforcement learning in an embodiment combining process supervision and GRPO algorithm; Figure 6 A schematic diagram of a training strategy combining process supervision and GRPO algorithm in an embodiment; Figure 7 A schematic diagram of a data example and model inference result based on fast and slow thinking in an embodiment; Figure 8 A structural block diagram of a brain-like scenario safety evaluation device combining process supervision and fast and slow thinking in an embodiment; Figure 9 An internal structure diagram of a computer device in an embodiment. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0019] The brain-like scenario safety evaluation method combining process supervision and fast and slow thinking provided by the embodiments of the present application can be applied in an application environment as shown in Figure 1 . In the application environment, a terminal communicates with a server through a network. A data storage system can store data required to be processed by the server. The data storage system can be integrated on the server, or placed on a cloud or other network server. The terminal can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart televisions, smart air conditioners, smart vehicle devices, etc. The portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server can be implemented by an independent server or a server cluster composed of multiple servers.

[0020] In an embodiment, as shown in Figure 2 , a brain-like scenario safety evaluation method combining process supervision and fast and slow thinking is provided. Taking the server in Figure 1 as an example, the method includes the following steps: S201: design a brain-like scenario safety awareness process and scenario safety level division criteria.

[0021] In the embodiments of the present application, first, inspired by brain cognition, a layered and progressive scene safety understanding process is constructed, and a scene safety level classification standard is agreed.

[0022] Specifically, as shown in Figure 3 In one embodiment of the present application, the brain-like scene safety cognition process includes entity perception, relationship extraction, overall understanding, and safety cognition.

[0023] S203: Construct a supervised fine-tuning data set and a reinforcement learning fine-tuning data set of process supervision paradigm based on the brain-like scene safety cognition process and the scene safety level classification standard.

[0024] In the embodiments of the present application, the powerful overall image understanding capability of GPT-4 is fully utilized to further supplement label information based on the target detection data set COCO and ImageNet, and attention is paid to converting into a supervised fine-tuning data set and a reinforcement learning fine-tuning data set of process supervision paradigm, as shown in the flow Figure 4

[0025] Specifically, in one embodiment of the present application, constructing a supervised fine-tuning data set and a reinforcement learning fine-tuning data set of process supervision paradigm based on the brain-like scene safety cognition process includes: S301: Collect image data based on a visual task, and generate corresponding image description data based on the image data.

[0026] S303: Manually annotate the image description data.

[0027] S305: Extract entities and corresponding entity relationships from the image description after manual annotation, and add target detection labels and relationship extraction labels based on the entities and corresponding entity relationships, respectively.

[0028] S307: Annotate safety level labels based on the scene safety level classification standard.

[0029] S309: Construct a supervised fine-tuning data set and a reinforcement learning fine-tuning data set of process supervision paradigm based on the image data, image description data, target detection labels, relationship extraction labels, and safety level labels.

[0030] ​In an embodiment of the present application, first, image data is collected from a target detection dataset, a relationship extraction dataset, and other visual task datasets, a multi-modal large language model is selected as a scene description generator, and the image data is input into the multi-modal large language model to output paragraph description data of each image. Then, the image description is checked, modified, or supplemented by an annotator, and the obtained data corresponds to the overall understanding link in the brain-like scene safety cognitive process. Then, entities and relationships between the entities are extracted from the image description after artificial annotation, and the target detection labels of the corresponding image data are supplemented based on the entities, and all target detection label data obtained correspond to the entity perception link in the brain-like scene safety cognitive process. At the same time, the relationship extraction labels of the corresponding image data are supplemented based on the relationships between the entities, and all relationship extraction label data obtained correspond to the relationship extraction link in the brain-like scene safety cognitive process. Then, the safety level labels of the image data are labeled based on the agreed scene safety level division standard, which corresponds to the safety cognition link in the brain-like scene safety cognitive process. Finally, according to the image, image description, target detection label, relationship extraction label, and safety level label, the data is converted into a supervised fine-tuning dataset of a multi-modal large language model and a reinforcement learning fine-tuning dataset of a process supervision paradigm.

[0031] In the embodiment, the dataset is constructed starting from the image description data, fully utilizing the powerful ability of the multi-modal large language model in describing images, and realizing efficient human-machine collaborative dataset construction combined with human modification and supplementation.

[0032] Specifically, in an embodiment of the present application, the constructing of the supervised fine-tuning dataset and the reinforcement learning fine-tuning dataset of the process supervision paradigm based on the image data, image description data, target detection label, relationship extraction label, and safety level label comprises: S401: design instructions of the supervised fine-tuning data and the reinforcement learning fine-tuning data of the process supervision paradigm.

[0033] S403: input the instructions as text in the supervised fine-tuning data and the reinforcement learning fine-tuning data of the process supervision paradigm respectively, and input the image data as image in the supervised fine-tuning data and the reinforcement learning fine-tuning data of the process supervision paradigm respectively.

[0034] S405: generate answers of the supervised fine-tuning data and the reinforcement learning fine-tuning data of the process supervision paradigm based on the image description data, the target detection label, the relationship extraction label, and the safety level label respectively.

[0035] In an embodiment of the present application, first, instructions for supervised fine-tuning data and instructions for reinforcement learning fine-tuning data of the process supervision paradigm are designed. Specifically, for the instructions for supervised fine-tuning data, the designed template is: "Please analyze the scene in the image step by step and give a safety level. For target detection, the format of the result should be: [{ 'position': [x1, y1, x2, y2], 'category': target category},...]. For relationship detection, the format of the result should be: [[subject, object, relationship],...]. For scene description, detailed and objective description is required. For safety level, the result should belong to [no risk, low risk, medium risk, high risk, extremely high risk]." For the instructions for reinforcement learning fine-tuning data of the process supervision paradigm, the designed template is: "Please analyze the scene in the image and give a safety level. The format of each step is: <think>Thinking process< / think> <answer>Results< / answer> . For target detection, the format of the result should be: [{ 'position': [x1, y1, x2, y2], 'category': target category},...]. For relationship detection, the format of the result should be: [[subject, object, relationship],...]. For scene description, detailed and objective description is required. For safety level, the result should belong to [no risk, low risk, medium risk, high risk, extremely high risk]." Wherein <think>And< / think> are used to mark the starting position and the ending position of the model thinking process respectively, <answer>And< / answer> are used to mark the starting position and the ending position of the result respectively.

[0036] Then, the instructions for supervised fine-tuning data are input as text in the supervised fine-tuning data, the instructions for reinforcement learning fine-tuning data of the process supervision paradigm are input as text in the reinforcement learning fine-tuning data of the process supervision paradigm, and the image data are input as images in the supervised fine-tuning data and the reinforcement learning fine-tuning data of the process supervision paradigm respectively.

[0037] Then, the text is used to describe the target detection coordinate box information, the relationship information, the image description and the safety level step by step in a certain format, and then the text is used as the model answer in the supervised fine-tuning data. Specifically, for the model answer of the supervised fine-tuning data, the designed format template is: "Step 1: [{ 'position': [ , , , ], 'category': },...]; Step 2: [[ , , ],...]; Step 3: ; Step 4: ." Wherein and respectively represent the horizontal and vertical coordinates of the upper left corner of the detection frame, and respectively represent the horizontal and vertical coordinates of the lower right corner of the detection frame, represents the category to which the target corresponding to the detection frame belongs, and respectively represent the subject target and the object target, represents the relationship between the subject target and , represents the scene description paragraph, represents the security level of the scene.

[0038] At the same time, the text is described in a certain format step by step in sequence to describe the target detection coordinate frame information, relationship information, image description and security level, and the starting position and ending position of each step are marked by <answer>And< / answer> respectively, and then the text is used as the model answer in the reinforcement learning fine-tuning data of the process supervision paradigm. Specifically, for the model answer of the reinforcement learning fine-tuning data of the process supervision paradigm, the designed format template is: "1、 <answer>[{ 'position': [ , , , ], 'category': }…]< / answer> 2、 <answer> [[ ， ， ],…]< / answer> 3、 <answer> < / answer> 4、 <answer> < / answer> ".

[0039] S205: Cold start the open source pre-training multi-modal large language model based on the supervision fine-tuning data set to obtain a supervision fine-tuning model.

[0040] In the embodiments of the present application, according to the parameter quantity, language type and GPU resource configuration, a suitable open source pre-training multi-modal large language model is selected, and the open source pre-training multi-modal large language model is supervised and fine-tuned in a self-recurrent form according to the supervision fine-tuning data to obtain a supervision fine-tuning model. The loss function used is:

[0041] wherein, is the predicted next token, is the sequence of all tokens before this in the model answer, is the token of the instruction input to the model, T is the length of the sequence, is the parameter of the supervision fine-tuning model, is the model prediction probability, denotes the concatenation operation, denotes the encoding of the input image using the image encoder of the multi-modal large language model.

[0042] S207: Reinforcement learning training is performed on the supervised fine-tuning model based on the reinforcement learning fine-tuning data set of the process supervision paradigm.

[0043] In an embodiment of the present application, the supervised fine-tuning model is further trained using the reinforcement learning fine-tuning data set of the process supervision paradigm and the GRPO algorithm based on process supervision, as shown in FIG. 7, which is a flowchart of the reinforcement learning training process. Figure 5 The training strategy of the GRPO algorithm based on process supervision is shown in detail in FIG. 8. Figure 6

[0044] In an embodiment of the present application, the reinforcement learning training performed on the supervised fine-tuning model based on the reinforcement learning fine-tuning data set of the process supervision paradigm includes: S501: The supervised fine-tuning model is copied into two models, one as a reference model with frozen parameters and the other as a policy model with trainable parameters.

[0045] S503: The text input and image input in the reinforcement learning fine-tuning data set of the process supervision paradigm are input into the policy model together to generate initial answers.

[0046] S505: A verifiable reward is designed, the reward value of each step of each initial answer is calculated based on the verifiable reward, and the advantage value of each token in the result part of each step is calculated based on the reward value.

[0047] S507: The KL divergence between the reference model and the policy model is calculated.

[0048] S509: The loss value is calculated based on the advantage value and the KL divergence, the policy model is trained based on the loss value, and the parameters are updated.

[0049] In an embodiment of the present application, the supervised fine-tuning model is copied into two models, one as a reference model with frozen parameters and the other as a policy model with trainable parameters. Then, the text input and image input in the reinforcement learning fine-tuning data set of the process supervision paradigm are input into the policy model together to sample multiple different initial answers. Then, a verifiable reward is designed, the reward value of each step of each initial answer is calculated based on the verifiable reward, and the advantage value of each token in the result part of each step is calculated based on the reward value, represented as:

[0050] wherein, represents the advantage value of each token in the result part of each step, represents the reward value of each step of each initial answer, ​token representing the need to calculate the advantage value, which token needs to come from the result part in each step, for calculating the step where the input token is located. token representing the step where the token is located and other subsequent steps, according to the cognitive process, the order of steps in the model answer is .

[0051] Then, the KL divergence between the strategy model and the reference model is calculated, denoted as:

[0052] wherein, represents the strategy model, represents the reference model, represents the i-th token in the j-th answer of the model, represents the i-th token in the j-th answer of the model, represents the input of the model (including the text instruction and the image), represents the other token sequence before the i-th token in the j-th answer of the model. Then, the loss value is calculated based on the advantage value and the KL divergence, denoted as:

[0053]

[0054] wherein, represents the number of answers generated by sampling, represents the old strategy model, and is a hyperparameter, is used for gradient clipping, which constrains the value of between and to avoid excessive updating of the model. According to the loss value, the strategy model is trained, and the parameters are updated.

[0055] In an embodiment of the present application, the designable reward comprises: S601: Designing a verifiable reward for the target detection result based on the predicted target detection result and the real target detection label.

[0056] S603: Designing a verifiable reward for the relationship extraction result based on the predicted target relationship and the real relationship extraction label.

[0057] S605: Designing a verifiable reward for the overall understanding result based on the predicted image description and the real image description.

[0058] ​​​S607: Designing the verifiable reward for the safety awareness result based on the predicted scene safety level and the real safety level label.

[0059] In an embodiment of the present application, the verifiable reward for the target detection result is designed as:

[0060] wherein, and represent the category space of the model-predicted target detection result and the category space of the real target detection label, respectively, is the number of the detection boxes in the model-predicted detection result belonging to the category , represents the result of the intersection-over-union matching between the model-predicted detection box of the category and the real detection box of the corresponding category under the threshold value of 0.5. represents the number of the prediction detection boxes of the category that can be successfully matched with the real detection box of the label, is the number of the detection boxes of the category in the label, and are parameters for controlling the false positive penalty intensity and the false negative penalty intensity, respectively, represents the format reward, if the format of the current step satisfies <think> …< / think> , <answer> …< / answer> then , otherwise .

[0061] The verifiable reward for the relationship extraction result is designed as:

[0062] wherein, represents the target category space constituted by the subject and the object in the model-predicted target relationship information, represents all the model-predicted target relationship information, represents all the real target relationship information in the label, is used to calculate the recall rate, is used to calculate the average recall rate of each relationship category.

[0063] The verifiable reward for the overall understanding result is designed as:

[0064] wherein, is according to the image description predicted by the model and the real image description to calculate the CIDEr-D score, representing the target class space extracted from the image description text predicted by the model, representing the target relationship information extracted from the image description text predicted by the model, representing the target relationship information predicted by the model in the target relationship extraction step.

[0065] The verifiable reward for the safety cognition result is designed as:

[0066] wherein, represents the scene safety level predicted by the model, represents the real scene safety level.

[0067] In this embodiment, by designing different forms of verifiable reward functions according to the task characteristics of different steps in the scene safety understanding process, a supervision signal is provided to the result of each step, so as to realize process supervision reinforcement learning and further improve the performance of the model.

[0068] In an embodiment of the present application, the calculation of the reward value of each step of each initial answer based on the verifiable reward comprises: S701: Each initial answer is segmented by steps, corresponding to the brain-like scene safety cognition process.

[0069] S703: Based on the corresponding model prediction result, the corresponding reward value is calculated respectively to obtain a reward set.

[0070] S705: Each step reward in the reward set is normalized to obtain the final reward value.

[0071] In an embodiment of the present application, first, each initial answer is segmented by steps, corresponding to the target detection, relationship extraction, overall understanding and safety cognition of the brain-like scene safety cognition process. Then, the corresponding model prediction result is extracted, and the reward of each step is calculated according to the prediction result of each step, the corresponding label and the calculation method of each reward, and the obtained all reward set is represented as:

[0072] wherein, is the number of generated answers.

[0073] Then, according to the reward result, each step reward in each answer is normalized to obtain the final reward, which is represented as:

[0074] wherein, and respectively for calculating the mean and variance.

[0075] S209: input the scene image to be evaluated into the supervised fine-tuning model after training, to obtain the scene safety level.

[0076] In the embodiments of the present application, the scene image to be evaluated is input into the trained model, and the model evaluates the scene safety of the test image in combination with fast and slow thinking to obtain the scene safety level, as shown in Figure 7 For each piece of data, only the result of each step is provided when constructing the label, and is placed between <answer>With< / answer> as a supervision signal during training. When the image is input into the finally trained model, the model can first perform slow thinking based on the prior knowledge built in the model before outputting the result of each step, and the thinking process is reflected between <think>With< / think> . After the thinking is completed, the result of this step is output to <answer>With< / answer> . Since there is no label information of the thinking process, no supervision signal is provided for the thinking process, maintaining the high generalization reasoning ability of the model, thereby encouraging the model to think flexibly to obtain the target desired by humans.

[0077] In the above brain-like scene safety evaluation method combining process supervision and fast and slow thinking, first, a brain-like scene safety cognitive process and a scene safety level division standard are designed; then, a supervised fine-tuning data set and a reinforcement learning fine-tuning data set of the process supervision paradigm are constructed based on the brain-like scene safety cognitive process and the scene safety level division standard; then, the open-source pre-trained multi-modal large language model is cold-started based on the supervised fine-tuning data set to obtain a supervised fine-tuning model; then, the supervised fine-tuning model is trained based on the reinforcement learning fine-tuning data set of the process supervision paradigm; finally, the scene image to be evaluated is input into the supervised fine-tuning model after training to obtain the scene safety level. That is, inspired by human-level cognitive mechanisms, the scene safety understanding task is divided into multiple key steps that progress layer by layer, the reasoning process of the model is aligned with the human thinking process on a macro level, and the model first performs deep slow thinking in each step. Through the paradigm of process supervision, the model realizes multiple slow thinking, and the supervision object is the conclusion of each slow thinking, corresponding to multiple key links in the human-level cognitive process, so that the model is aligned with the human cognitive process on a macro level in the reasoning process, and the result of this step is output. In the training process, the slow thinking process is ignored, the model fully utilizes the prior knowledge possessed by itself to reason flexibly, effectively avoids the problem of insufficient or excessive reasoning of the model when facing complex reasoning tasks, reduces the occurrence of hallucinations, maintains the flexibility of the thinking process, and improves the explainability of the result of each step.

[0078] It should be understood that although the steps in the flowcharts involved in the embodiments described above are shown in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, at least some of the steps in the flowcharts involved in the embodiments described above can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be alternately or alternately executed with at least part of other steps or steps or stages in other steps.

[0079] Based on the same inventive concept, the embodiments of the present application also provide a combination process supervision and fast and slow thinking brain-like scene safety evaluation device for implementing the above-mentioned combination process supervision and fast and slow thinking brain-like scene safety evaluation method. The implementation scheme for solving problems provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more combination process supervision and fast and slow thinking brain-like scene safety evaluation device embodiments provided below can be referred to the limitations of the combination process supervision and fast and slow thinking brain-like scene safety evaluation method in the above, which will not be repeated here.

[0080] In one embodiment, as shown in Figure 8 A combination process supervision and fast and slow thinking brain-like scene safety evaluation device 800 is provided, comprising: a brain-like scene safety cognitive process and grade division standard design module 801, a data set construction module 803, a model cold start module 805, a model reinforcement learning training module 807, and a scene safety evaluation module 809, wherein: The brain-like scene safety cognitive process and grade division standard design module 801 is used to design a brain-like scene safety cognitive process and a scene safety grade division standard.

[0081] The data set construction module 803 is used to construct a supervised fine-tuning data set and a reinforcement learning fine-tuning data set of a process supervision paradigm based on the brain-like scene safety cognitive process and the scene safety grade division standard.

[0082] The model cold start module 805 is used to cold start an open source pre-training multi-modal large language model based on the supervised fine-tuning data set to obtain a supervised fine-tuning model.

[0083] The model reinforcement learning training module 807 is used to perform reinforcement learning training on the supervised fine-tuning model based on the reinforcement learning fine-tuning data set of the process supervision paradigm.

[0084] The scene safety evaluation module 809 is configured to input a scene image to be evaluated into the supervised fine-tuning model after training to obtain a scene safety level.

[0085] In an embodiment of the present application, the brain-like scene safety cognitive process includes entity perception, relationship extraction, overall understanding, and safety cognition.

[0086] In an embodiment of the present application, the dataset construction module is further configured to: collect image data based on a visual task, and generate corresponding image description data based on the image data; manually annotate the image description data; extract entities and corresponding entity relationships from the image description data after manual annotation, and add target detection labels and relationship extraction labels based on the entities and corresponding entity relationships, respectively; label safety level labels based on scene safety level division criteria; construct a supervised fine-tuning dataset and a reinforcement learning fine-tuning dataset of a process supervision paradigm based on the image data, image description data, target detection labels, relationship extraction labels, and safety level labels.

[0087] In an embodiment of the present application, the dataset construction module is further configured to: design instructions of the supervised fine-tuning data and the reinforcement learning fine-tuning data of the process supervision paradigm; input the instructions as text in the supervised fine-tuning data and the reinforcement learning fine-tuning data of the process supervision paradigm, respectively, and input the image data as images in the supervised fine-tuning data and the reinforcement learning fine-tuning data of the process supervision paradigm, respectively; generate answers of the supervised fine-tuning data and the reinforcement learning fine-tuning data of the process supervision paradigm based on the image description data, target detection labels, relationship extraction labels, and safety level labels, respectively.

[0088] In an embodiment of the present application, the model reinforcement learning training module is further configured to: copy the supervised fine-tuning model, one as a reference model and one as a policy model; input the text and image inputs in the reinforcement learning fine-tuning dataset of the process supervision paradigm into the policy model together to generate initial answers; design a verifiable reward, calculate a reward value of each step of each initial answer based on the verifiable reward, and calculate an advantage value of each token of a result part of each step based on the reward value; calculate a KL divergence between the reference model and the policy model; Calculate a loss value based on the advantage value and the KL divergence, train the policy model based on the loss value, and update the parameters.

[0089] In an embodiment of the present application, the model reinforcement learning training module is further configured to: Design a verifiable reward for the target detection result based on the predicted target detection result and the real target detection label; Design a verifiable reward for the relationship extraction result based on the predicted target relationship and the real relationship extraction label; Design a verifiable reward for the overall understanding result based on the predicted image description and the real image description; Design a verifiable reward for the safety cognition result based on the predicted scene safety level and the real safety level label.

[0090] In an embodiment of the present application, the model reinforcement learning training module is further configured to: Divide each initial answer into steps, corresponding to the brain-like scene safety cognition process; Calculate the corresponding reward value based on the corresponding model prediction result, and obtain a reward set; Normalize each step reward in the reward set to obtain the final reward value.

[0091] The above-mentioned various modules in the brain-like scene safety evaluation device combining process supervision and fast and slow thinking can be realized by software, hardware and combinations thereof, in whole or in part. The above-mentioned various modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to the above-mentioned various modules.

[0092] In an embodiment, a computer device is provided, which can be a terminal, and its internal structure diagram can be as shown in Figure 9As shown in the figure. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used for wired or wireless communication with external terminals. Wireless mode can be achieved through WIFI, mobile cellular network, NFC (near field communication) or other technologies. The computer program is executed by the processor to implement a brain-like scene security evaluation method combining process supervision and fast and slow thinking. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device, or an external keyboard, touchpad or mouse, etc.

[0093] Those skilled in the art can understand that, Figure 9 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0094] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps in the above method embodiments.

[0095] In one embodiment, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the steps in the above method embodiments.

[0096] In one embodiment, a computer program product is provided, including a computer program, and the computer program is executed by a processor to implement the steps in the above method embodiments.

[0097] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties.

[0098] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0099] Any combination of the technical features of the above embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combination of the technical features does not exist, it should be considered as the scope of the present application.

[0100] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A brain-like scenario safety assessment method combining process supervision and fast and slow thinking, characterized by: The method comprises: Design brain-like scenario safety cognitive processes and scenario safety level classification standards; Based on the brain-like scene safety cognitive process and scene safety level classification standard, a supervised fine-tuning dataset and a reinforcement learning fine-tuning dataset of the process supervision paradigm are constructed; Cold-starting the open-source pre-trained multimodal large language model based on the supervised fine-tuning dataset to obtain a supervised fine-tuning model; Performing reinforcement learning training on the supervised fine-tuning model based on a reinforcement learning fine-tuning dataset of the process supervision paradigm; The image of the scene to be evaluated is input into the trained supervised fine-tuning model to obtain the scene safety level.

2. The brain-like scenario safety assessment method combining process supervision and fast and slow thinking according to claim 1 is characterized in that: The brain-like scenario safety cognition process includes entity perception, relationship extraction, overall understanding, and safety cognition.

3. The brain-like scenario safety assessment method combining process supervision and fast and slow thinking according to claim 1 is characterized in that: The supervised fine-tuning dataset and the reinforcement learning fine-tuning dataset based on the brain-like scenario safety cognitive process and the process supervision paradigm include: Collect image data based on a visual task, and generate corresponding image description data based on the image data; Manually labeling the image description data; Extracting entities and corresponding entity relationships from the manually annotated image description, and adding target detection labels and relationship extraction labels based on the entities and corresponding entity relationships; Label security levels based on scenario security level classification standards; Based on the image data, image description data, target detection labels, relationship extraction labels and security level labels, a supervised fine-tuning dataset and a reinforcement learning fine-tuning dataset of a process supervision paradigm are constructed.

4. The brain-like scenario safety assessment method combining process supervision and fast and slow thinking according to claim 3 is characterized in that: The construction of a supervised fine-tuning dataset and a reinforcement learning fine-tuning dataset of a process supervision paradigm based on the image data, image description data, target detection labels, relationship extraction labels, and security level labels includes: Design supervised fine-tuning data and instructions for reinforcement learning fine-tuning data for the process supervision paradigm; The instructions are used as text inputs in supervised fine-tuning data and reinforcement learning fine-tuning data of a process-supervised paradigm, and the image data are used as image inputs in supervised fine-tuning data and reinforcement learning fine-tuning data of a process-supervised paradigm; Based on image description data, object detection labels, relationship extraction labels and security level labels, answers are generated for supervised fine-tuning data and reinforcement learning fine-tuning data of the process supervision paradigm respectively.

5. The brain-like scenario safety assessment method combining process supervision and fast and slow thinking according to claim 1 is characterized in that: The reinforcement learning fine-tuning dataset based on the process supervision paradigm is used to perform reinforcement learning training on the supervised fine-tuning model, comprising: Duplicate the supervised fine-tuning model, one as a reference model and one as a policy model; Inputting text input and image input from the reinforcement learning fine-tuning dataset of the process supervision paradigm into the policy model to generate an initial answer; Design a verifiable reward, calculate a reward value for each step of each initial answer based on the verifiable reward, and calculate an advantage value for each token in the result part of each step based on the reward value; Calculating the KL divergence between the reference model and the policy model; A loss value is calculated based on the advantage value and the KL divergence, and the strategy model is trained based on the loss value to update parameters.

6. The brain-like scenario safety assessment method combining process supervision and fast and slow thinking according to claim 5 is characterized in that: The Verifiable Design Rewards include: Design verifiable rewards for target detection results based on predicted target detection results and true target detection labels; Design verifiable rewards for relation extraction results based on predicted target relations and true relation extraction labels; Design verifiable rewards for overall understanding based on predicted and ground-truth image descriptions; Design verifiable rewards for safety cognition results based on the predicted scenario safety level and the actual safety level label.

7. The brain-like scenario safety assessment method combining process supervision and fast and slow thinking according to claim 5 is characterized in that: The reward value of each step of calculating each initial answer based on the verifiable reward includes: Each initial answer is divided into steps corresponding to the safety cognitive process of the brain-like scenario; Calculate the corresponding reward values ​​based on the corresponding model prediction results to obtain a reward set; Normalize the rewards for each step in the reward set to get the final reward value.

8. A brain-like scenario safety assessment device that combines process supervision and fast and slow thinking, characterized by: The device comprises: A module for designing the cognitive process and grading standards for brain-like scenario safety, used to design the cognitive process and grading standards for brain-like scenario safety; A dataset construction module is used to construct a supervised fine-tuning dataset and a reinforcement learning fine-tuning dataset of the process supervision paradigm based on the brain-like scene safety cognitive process and scene safety level classification standard; A model cold start module, configured to cold start the open source pre-trained multimodal large language model based on the supervised fine-tuning dataset to obtain a supervised fine-tuning model; A model reinforcement learning training module, configured to perform reinforcement learning training on the supervised fine-tuning model based on a reinforcement learning fine-tuning dataset of the process supervision paradigm; The scene safety assessment module is used to input the scene image to be evaluated into the trained supervised fine-tuning model to obtain the scene safety level.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images