A security and protection foundation big model based on preference reinforcement learning and a training method thereof

Abstract

The application relates to the technical field of model training, and particularly discloses a security foundation big model based on preference reinforcement learning and a training method, which comprises the following steps: a processing module is used to obtain all positive samples and all negative samples of an input video group based on the portrait description of all videos of the input video group; a training module is used to obtain all comparison sample pairs based on all positive samples, all negative samples and a preset unlabeled video group of the input video group; a calculation module is used to obtain the iterative sorting of all comparison sample pairs based on all comparison sample pairs and a preset preference learning technology; and an iteration module is used to obtain the iterative screening straight line of the security foundation big model based on the iterative sorting of all comparison sample pairs, so that the security foundation big model is obtained. The application can train a model which can accurately deduce the correct portrait description of an input video by using a small amount of training data.

Description

Technical Field

[0001] This invention relates to the field of model training technology, and in particular to a large-scale security basic model and training method based on preference reinforcement learning. Background Technology

[0002] Currently, security surveillance video data is massive, with city-level cameras generating petabytes of data daily. However, high-quality labeled data is scarce, and manual labeling is extremely costly. Traditional fully supervised learning relies on large-scale labeled data, making it difficult for security companies to quickly deploy models. Model performance is positively correlated with the amount of labeled data, but labeled resources are limited in security scenarios, requiring high-precision inference with limited data. Therefore, how to train a model with limited training data that can accurately derive correct portrait descriptions of people from input videos is a pressing issue in the field of model training technology.

[0003] However, to date, there is no security basic model or training method based on preference reinforcement learning that can train a model that can accurately derive the correct portrait description of the person in the input video using a small amount of training data.

[0004] Therefore, this invention proposes a large-scale security basic model and training method based on preference reinforcement learning. Summary of the Invention

[0005] This invention provides a security foundation model and training method based on preference reinforcement learning. It obtains all positive and negative samples from the input video group based on the portrait descriptions of all videos in the input video group. This facilitates the subsequent generation of a high-quality and low-quality security model based on these samples. Using the high-quality and low-quality models and a pre-defined unlabeled video group, it obtains all comparison sample pairs. Based on these comparison sample pairs and a pre-defined preference learning technique, it obtains the loss value and priority iteration value for each comparison sample pair. This quantifies the degree to which the pre-trained model violates the preference relationship of each comparison sample pair and quantifies the priority of each comparison sample pair's corresponding video as training iteration input for the pre-trained model. Furthermore, based on the priority iteration values ​​of all comparison sample pairs, it obtains the iterative ranking of all comparison sample pairs. Based on this iterative ranking, it obtains the iterative filtering line for the security foundation model. This line, which can be plotted and processed to filter comparison sample pairs, ultimately yields the security foundation model. This achieves a model trained with a small amount of training data that can accurately derive the correct portrait descriptions of input videos.

[0006] This invention provides a large-scale security foundation model based on preference reinforcement learning, comprising:

[0007] The processing module is used to obtain the portrait descriptions of people in all videos of the input video group based on the pre-trained model and the input video group, and to obtain all positive samples and all negative samples of the input video group based on the portrait descriptions of people in all videos of the input video group.

[0008] The training module is used to obtain a high-quality security model and a low-quality security model based on all positive and negative samples of the input video group, and to obtain all comparison sample pairs based on the high-quality security model, the low-quality security model and the preset unlabeled video group.

[0009] The calculation module is used to obtain the loss value and priority iteration value of all comparison sample pairs based on all comparison sample pairs and the preset preference learning technique, and to obtain the iterative ranking of all comparison sample pairs based on the priority iteration value of all comparison sample pairs.

[0010] The iterative module is used to obtain the iterative screening line of the security basic model based on the iterative sorting of all comparison sample pairs, and to obtain the security basic model based on the iterative screening line of the security basic model.

[0011] The preferred security foundation model based on preference reinforcement learning, including the following processing module:

[0012] The first processing submodule is used to sequentially input each video of the input video group into the pre-trained model to obtain the portrait description of the person in each video of the input video group;

[0013] The second processing submodule is used to obtain all positive samples and all negative samples of the input video group based on the portrait descriptions of all videos in the input video group.

[0014] The preferred security foundation model based on preference reinforcement learning, the second processing submodule, includes:

[0015] The first processing unit is used to input the portrait description of each video in the input video group into a preset large language model to obtain the reasoning action result of each video in the input video group.

[0016] The second processing unit is used to determine whether the reasoning action result of each video in the input video group is the same as the real label. If so, the corresponding video in the input video group is treated as a positive sample of the input video group; otherwise, the corresponding video in the input video group is treated as a negative sample of the input video group.

[0017] The preferred training module for a large-scale security model based on preference reinforcement learning includes:

[0018] The first training submodule is used to obtain a high-quality security model by using only all positive samples of the input video group as training samples of the pre-trained model, and to obtain a low-quality security model by using only all negative samples of the input video group as training samples of the pre-trained model.

[0019] The second training submodule is used to obtain all comparison sample pairs based on the high-quality security model, the low-quality security model, and the preset unlabeled video group.

[0020] The preferred security foundation model based on preference reinforcement learning, the second training submodule, includes:

[0021] The first training unit is used to receive all unlabeled videos from a preset unlabeled video group, input each unlabeled video from the preset unlabeled video group into the high-quality security model, obtain a high-quality portrait description of each unlabeled video from the preset unlabeled video group, and input each unlabeled video from the preset unlabeled video group into the low-quality security model, obtain a low-quality portrait description of each unlabeled video from the preset unlabeled video group.

[0022] The second training unit is used to treat the high-quality and low-quality portrait descriptions of each unlabeled video in the preset unlabeled video group as a comparison sample pair.

[0023] The preferred large-scale security model based on preference reinforcement learning includes the following computation module:

[0024] The first calculation submodule is used to obtain the high-quality description score and the low-quality description score of each comparison sample pair based on all comparison sample pairs and the preset preference learning technique, and to obtain the loss value of each comparison sample pair based on the high-quality description score and the low-quality description score of each comparison sample pair.

[0025] The second calculation submodule is used to obtain the priority iteration value of each comparison sample pair based on the loss value of all comparison sample pairs, and to obtain the iterative ranking of all comparison sample pairs based on the priority iteration values ​​of all comparison sample pairs.

[0026] The preferred security foundation model based on preference reinforcement learning, the second computational submodule, includes:

[0027] The first calculation unit is used to obtain the priority iteration value for each comparison sample pair based on the loss value, high-quality description score, and low-quality description score of all comparison sample pairs.

[0028] The second calculation unit is used to sort all comparison sample pairs in ascending order of priority iteration value to obtain the iterative sort of all comparison sample pairs.

[0029] The preferred security foundation model based on preference reinforcement learning, including the iterative module, comprises:

[0030] The first iteration submodule is used to take the priority iteration value of all comparison sample pairs as the x-axis value and the loss value of the corresponding comparison sample pair as the y-axis value to obtain all iteration observation points. It also takes two adjacent comparison sample pairs in the iteration sorting of all comparison sample pairs as a comparison sample pair group, and connects the iteration observation points corresponding to the two comparison sample pairs in each comparison sample pair group to obtain the iteration observation line segment of each comparison sample pair group.

[0031] The second iteration submodule is used to define the ordinal numbers of the iterative observation line segments of all comparison sample pairs in ascending order of slope, starting from 1, to obtain the ordinal definition results of the iterative observation line segments of all comparison sample pairs. Based on the ordinal definition results of the iterative observation line segments of all comparison sample pairs, the iterative screening line segments of the security basic large model are obtained. The two ends of the iterative screening line segments of the security basic large model are extended to obtain the iterative screening line of the security basic large model.

[0032] The third iteration submodule is used to iteratively filter straight lines based on the security basic model to obtain the security basic model.

[0033] The preferred security foundation model based on preference reinforcement learning, third iteration submodule, includes:

[0034] The iteration unit is used to treat the iteration observation point with the smallest ordinate among all iteration observation points as the marker point, and the line passing through the marker point and perpendicular to the iteration screening line of the security basic model as the iteration screening vertical line of the security basic model. The comparison sample pairs corresponding to all iteration observation points whose positions are simultaneously below the iteration screening line and the iteration screening vertical line of the security basic model are treated as the screening comparison sample pairs, and the unlabeled videos corresponding to the screening comparison sample pairs are treated as the screening iteration videos.

[0035] The modeling unit is used to use all the selected and iterated videos as training samples for the pre-trained model to obtain a large basic security model.

[0036] This invention provides a method for training a large-scale security basic model based on preference reinforcement learning, which is applied to train any one of the large-scale security basic models based on preference reinforcement learning in Examples 1 to 9, including:

[0037] S1: Based on the pre-trained model and the input video group, obtain the portrait descriptions of the people in all videos of the input video group, and based on the portrait descriptions of the people in all videos of the input video group, obtain all positive samples and all negative samples of the input video group.

[0038] S2: Based on all positive and negative samples of the input video group, obtain the high-quality security model and the low-quality security model, and based on the high-quality security model, the low-quality security model and the preset unlabeled video group, obtain all comparison sample pairs;

[0039] S3: Based on all comparison sample pairs and the preset preference learning technique, obtain the loss value and priority iteration value of all comparison sample pairs, and based on the priority iteration value of all comparison sample pairs, obtain the iterative ranking of all comparison sample pairs;

[0040] S4: Based on the iterative sorting of all comparison sample pairs, obtain the iterative screening line of the security basic large model, and based on the iterative screening line of the security basic large model, obtain the security basic large model.

[0041] The beneficial effects of this invention compared to existing technologies are as follows: Based on the portrait descriptions of all videos in the input video group, all positive and negative samples of the input video group are obtained. This facilitates the subsequent acquisition of a high-quality security model and a low-quality security model based on all positive and negative samples of the input video group. Based on the high-quality and low-quality security models and the preset unlabeled video group, all comparison sample pairs are obtained. Based on all comparison sample pairs and the preset preference learning technique, the loss value and priority iteration value of all comparison sample pairs are obtained. This quantifies the degree of violation of the preference relationship of the pre-trained model for each comparison sample pair and quantifies the priority of the corresponding video of each comparison sample pair as the training iteration input of the pre-trained model. Furthermore, based on the priority iteration value of all comparison sample pairs, the iterative ranking of all comparison sample pairs is obtained. Based on the iterative ranking of all comparison sample pairs, the iterative screening line of the basic security model is obtained. The line can be plotted and processed to select the comparison sample pairs. Finally, based on the iterative screening line of the basic security model, the basic security model is obtained. This realizes a model that can accurately derive the correct portrait description of the input video using a small amount of training data.

[0042] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written documents of this application.

[0043] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0044] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0045] Figure 1This is a schematic diagram of a basic security model based on preference reinforcement learning in an embodiment of the present invention;

[0046] Figure 2 This is a flowchart of a security basic large model training method based on preference reinforcement learning in an embodiment of the present invention. Detailed Implementation

[0047] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0048] Example 1: This invention provides a large-scale security foundation model based on preference reinforcement learning, referencing... Figure 1 ,include:

[0049] The processing module is used to obtain the portrait descriptions of people in all videos of the input video group based on the pre-trained model and the input video group, and to obtain all positive samples and all negative samples of the input video group based on the portrait descriptions of people in all videos of the input video group.

[0050] The training module is used to obtain a high-quality security model and a low-quality security model based on all positive and negative samples of the input video group, and to obtain all comparison sample pairs based on the high-quality security model, the low-quality security model and the preset unlabeled video group.

[0051] The calculation module is used to obtain the loss value and priority iteration value of all comparison sample pairs based on all comparison sample pairs and the preset preference learning technique, and to obtain the iterative ranking of all comparison sample pairs based on the priority iteration value of all comparison sample pairs.

[0052] The iterative module is used to obtain the iterative screening line of the security basic model based on the iterative sorting of all comparison sample pairs, and to obtain the security basic model based on the iterative screening line of the security basic model.

[0053] In this embodiment, the pre-trained model is a deep learning model that has completed initial training on a large-scale general dataset. It can take video as input to the model and then output a portrait description of the person in the video, such as "a man wearing glasses is in the classroom and is raising his hand to speak".

[0054] In this embodiment, the input video group is a pre-set combination containing multiple videos, and each video in the input video group has been manually labeled and has real tags.

[0055] In this embodiment, the portrait description is the process of systematically expressing the identity attributes, behaviors, and scene-related information of a person in a video or image through structured or natural language.

[0056] In this embodiment, positive samples are the correctly inferred portrait descriptions among the portrait descriptions of all videos in the input video group.

[0057] In this embodiment, negative samples are the incorrectly inferred portrait descriptions among all the portrait descriptions in the input video group.

[0058] In this embodiment, the high-quality security model is a model that can learn to generate portrait descriptions of people who can deduce correct actions by using only all positive samples of the input video group as training samples for the pre-trained model (that is, using the video corresponding to each positive sample of the input video group as training input and the corresponding positive sample of the input video group as training output). This model can output portrait descriptions of people in the video as long as the video is input.

[0059] In this embodiment, the security inferiority model is a model that can simulate the generation of character portrait descriptions that lead to erroneous reasoning (such as missing key clues or introducing interference / erroneous information) by using only all negative samples of the input video group as training samples of the pre-trained model (that is, using the video corresponding to each negative sample of the input video group as training input and the corresponding negative sample of the input video group as training output). This model can output character portrait descriptions of the video as input.

[0060] In this embodiment, the preset unlabeled video group is a pre-set combination of multiple videos that have not been manually labeled.

[0061] In this embodiment, the comparison sample pair is a comparison combination of high-quality and low-quality portrait descriptions of people in each unlabeled video of a preset unlabeled video group.

[0062] In this embodiment, the preset preference learning technology is a pre-set, existing preference learning technology, such as preference relationship definition, scoring function design, etc.

[0063] In this embodiment, the loss value is the degree to which the pre-trained model violates the preference relationship for each pair of contrasting samples.

[0064] In this embodiment, the priority iteration value is a numerical value that characterizes the priority of each comparison sample to the corresponding video as the training iteration input of the pre-trained model.

[0065] In this embodiment, the iterative sorting is the sorting result obtained by sorting all comparison sample pairs according to the priority iteration value of all comparison sample pairs.

[0066] In this embodiment, the iterative screening line of the security basic large model is a line that can be used to screen out the comparison sample pairs.

[0067] In this embodiment, the basic security model is a basic artificial intelligence model designed specifically for security scenarios, which optimizes decision-making logic through preference reinforcement learning technology. This model can receive video outputs corresponding video portrait descriptions.

[0068] The beneficial effects of the above technology are as follows: Based on the portrait descriptions of all videos in the input video group, all positive and negative samples of the input video group are obtained, which facilitates the subsequent acquisition of high-quality and low-quality security models based on all positive and negative samples of the input video group. Based on the high-quality and low-quality security models and the preset unlabeled video group, all comparison sample pairs are obtained. Based on all comparison sample pairs and the preset preference learning technique, the loss value and priority iteration value of all comparison sample pairs are obtained. This realizes the quantification of the degree of violation of the preference relationship of the pre-trained model for each comparison sample pair and the quantification of the priority of the corresponding video of each comparison sample pair as the training iteration input of the pre-trained model. Then, based on the priority iteration value of all comparison sample pairs, the iterative ranking of all comparison sample pairs is obtained. Based on the iterative ranking of all comparison sample pairs, the iterative screening line of the basic security model is obtained. The line can be plotted and processed to select the comparison sample pairs. Finally, based on the iterative screening line of the basic security model, the basic security model is obtained. This realizes a model that can accurately derive the correct portrait description of the input video using a small amount of training data.

[0069] Example 2: Based on Example 1, the security fundamental model based on preference reinforcement learning includes the following processing module:

[0070] The first processing submodule is used to sequentially input each video of the input video group into the pre-trained model to obtain the portrait description of the person in each video of the input video group;

[0071] The second processing submodule is used to obtain all positive samples and all negative samples of the input video group based on the portrait descriptions of all videos in the input video group.

[0072] The beneficial effects of the above technology are: based on the portrait descriptions of the people in all videos of the input video group, all positive samples and all negative samples of the input video group can be obtained, realizing the construction of a high-discrimination training set with only a small amount of labeled data, which greatly reduces the labeling cost.

[0073] Example 3: Based on Example 2, the second processing submodule of the security foundation model based on preference reinforcement learning includes:

[0074] The first processing unit is used to input the portrait description of each video in the input video group into a preset large language model to obtain the reasoning action result of each video in the input video group.

[0075] The second processing unit is used to determine whether the reasoning action result of each video in the input video group is the same as the real label. If so, the corresponding video in the input video group is treated as a positive sample of the input video group; otherwise, the corresponding video in the input video group is treated as a negative sample of the input video group.

[0076] In this embodiment, the preset large language model is a pre-set model that can perform action reasoning on the portrait description of a person in the video (the input of the preset large language model, such as a man waving and there is a bus next to him) in order to obtain the reasoning action result of the video (the output of the preset large language model, such as a man waving to signal the bus to stop).

[0077] In this embodiment, the real label is the result of manual annotation of each video in the input video group in advance, such as a child waving to signal a bus to stop.

[0078] In this embodiment, the reasoning action result for each video in the same input video group is synonymous with the real label.

[0079] The beneficial effects of the above technology are as follows: based on the portrait descriptions of the people in all videos of the input video group, all positive samples and all negative samples of the input video group are obtained, and the method for determining all positive samples and all negative samples of the input video group is given in detail.

[0080] Example 4: Based on Example 1, the training module for a large-scale security basic model based on preference reinforcement learning includes:

[0081] The first training submodule is used to obtain a high-quality security model by using only all positive samples of the input video group as training samples of the pre-trained model, and to obtain a low-quality security model by using only all negative samples of the input video group as training samples of the pre-trained model.

[0082] The second training submodule is used to obtain all comparison sample pairs based on the high-quality security model, the low-quality security model, and the preset unlabeled video group.

[0083] The beneficial effects of the above technology are as follows: based on the high-quality security model, the low-quality security model, and the preset unlabeled video group, all comparison sample pairs are obtained, and the construction methods of the high-quality security model and the low-quality security model are given in detail.

[0084] Example 5: Based on Example 4, the second training submodule of the security foundation model based on preference reinforcement learning includes:

[0085] The first training unit is used to receive all unlabeled videos from a preset unlabeled video group, input each unlabeled video from the preset unlabeled video group into the high-quality security model, obtain a high-quality portrait description of each unlabeled video from the preset unlabeled video group, and input each unlabeled video from the preset unlabeled video group into the low-quality security model, obtain a low-quality portrait description of each unlabeled video from the preset unlabeled video group.

[0086] The second training unit is used to treat the high-quality and low-quality portrait descriptions of each unlabeled video in the preset unlabeled video group as a comparison sample pair.

[0087] The beneficial effects of the above technology are as follows: based on the high-quality security model, the low-quality security model, and the preset unlabeled video group, all comparison sample pairs are obtained, and the method for determining all comparison sample pairs is given in detail.

[0088] Example 6: Based on Example 1, a large-scale security fundamental model based on preference reinforcement learning, including a computation module, comprises:

[0089] The first calculation submodule is used to obtain the high-quality description score and the low-quality description score of each comparison sample pair based on all comparison sample pairs and the preset preference learning technique, and to obtain the loss value of each comparison sample pair based on the high-quality description score and the low-quality description score of each comparison sample pair.

[0090] The second calculation submodule is used to obtain the priority iteration value of each comparison sample pair based on the loss value of all comparison sample pairs, and to obtain the iterative ranking of all comparison sample pairs based on the priority iteration values ​​of all comparison sample pairs.

[0091] In this embodiment, the quality score of the high-quality description is the score assigned to the high-quality portrait description in each comparison sample pair based on a preset preference learning technique (existing scoring function design techniques, such as direct comparison scoring, implicit scoring based on similarity, etc.).

[0092] In this embodiment, the quality score of the poor description is the score assigned to the poor portrait description in each comparison sample pair based on a preset preference learning technique (existing scoring function design techniques, such as direct comparison scoring, implicit scoring based on similarity, etc.).

[0093] In this embodiment, based on the high-quality description score and low-quality description score of each comparison sample pair, the loss value of each comparison sample pair is obtained, which is:

[0094] ;

[0095] Where α is the loss value of the currently calculated comparison sample pair, and max is the function that takes the maximum value. The preset interval (meaning the quality score for a high-quality description must be at least higher than the quality score for a low-quality description) ), Assign a score to the quality of the description of the currently calculated comparison sample pairs. Assign a quality score to the poor description of the currently calculated comparison sample pair.

[0096] The beneficial effects of the above techniques are as follows: Based on all comparison sample pairs and the preset preference learning technique, the high-quality description score and low-quality description score of each comparison sample pair are obtained, which facilitates the calculation of the loss value of each comparison sample pair in the future. Based on the high-quality description score and low-quality description score of each comparison sample pair, the loss value of each comparison sample pair is obtained, thereby realizing the quantification of the degree of violation of the preference relationship of the pre-trained model on each comparison sample pair.

[0097] Example 7: Based on Example 6, the second computational submodule of the security fundamental model based on preference reinforcement learning includes:

[0098] The first calculation unit is used to obtain the priority iteration value for each comparison sample pair based on the loss value, high-quality description score, and low-quality description score of all comparison sample pairs.

[0099] The second calculation unit is used to sort all comparison sample pairs in ascending order of priority iteration value to obtain the iterative sort of all comparison sample pairs.

[0100] The beneficial effects of the above technology are as follows: Based on all comparison sample pairs and the preset preference learning technology, the priority iteration value of all comparison sample pairs is obtained, which realizes the quantification of the priority of the video corresponding to each comparison sample pair as the training iteration input of the pre-trained model. Then, based on the priority iteration value of all comparison sample pairs, the iteration ranking of all comparison sample pairs is obtained, which facilitates the construction of the iterative selection line of the subsequent security basic large model.

[0101] Example 8: Based on Example 5, an iterative module for a security fundamental model based on preference reinforcement learning includes:

[0102] The first iteration submodule is used to take the priority iteration value of all comparison sample pairs as the x-axis value and the loss value of the corresponding comparison sample pair as the y-axis value to obtain all iteration observation points. It also takes two adjacent comparison sample pairs in the iteration sorting of all comparison sample pairs as a comparison sample pair group, and connects the iteration observation points corresponding to the two comparison sample pairs in each comparison sample pair group to obtain the iteration observation line segment of each comparison sample pair group.

[0103] The second iteration submodule is used to define the ordinal numbers of the iterative observation line segments of all comparison sample pairs in ascending order of slope, starting from 1, to obtain the ordinal definition results of the iterative observation line segments of all comparison sample pairs. Based on the ordinal definition results of the iterative observation line segments of all comparison sample pairs, the iterative screening line segments of the security basic large model are obtained. The two ends of the iterative screening line segments of the security basic large model are extended to obtain the iterative screening line of the security basic large model.

[0104] The third iteration submodule is used to iteratively filter straight lines based on the security basic model to obtain the security basic model.

[0105] In this embodiment, based on the ordinal definition results of the iterative observation line segments of all comparison sample groups, the iteratively selected line segments of the security basic large model are obtained, namely:

[0106] Obtain the slope and coordinates of the midpoint (midpoint of the line segment) of the iterative observation line segment for each comparison sample group;

[0107] All selection points are obtained by taking the x-coordinate of the midpoint of the iterative observation line segment of all comparison sample groups as the x-coordinate, the y-coordinate of the midpoint of the iterative observation line segment of the corresponding comparison sample group as the y-coordinate, and the slope of the iterative observation line segment of the corresponding comparison sample group as the y-coordinate.

[0108] Using the selection point corresponding to the iterative observation line segment of the comparison sample group with an ordinal definition result of 1 as the center of the sphere, and the length of the iterative observation line segment of the comparison sample group with an ordinal definition result of 1 as the radius, the spherical region of the iterative observation line segment of the comparison sample group with an ordinal definition result of 1 is obtained.

[0109] Determine whether the quotient between the number of remaining screening points within the spherical region of the iterative observation line segment of the comparison sample group with ordinal definition result 1 and the total number of all screening points is greater than a preset threshold (e.g., 80%). If yes, then the iterative observation line segment of the comparison sample group with ordinal definition result 1 is taken as the iterative screening line segment of the security basic large model. Otherwise, continue to determine whether the quotient between the number of remaining screening points within the spherical region of the iterative observation line segment of the comparison sample group with ordinal definition result 2 and the total number of all screening points is greater than a preset threshold (e.g., 80%). If yes, then the iterative observation line segment of the comparison sample group with ordinal definition result 2 is taken as the iterative screening line segment of the security basic large model. Otherwise, continue to repeat the above process until the iterative screening line segment of the security basic large model is obtained. (If the quotient between the number of remaining screening points within the spherical region of the iterative observation line segment of all comparison sample groups and the total number of all screening points is not greater than the preset threshold (e.g., 80%), then the iterative observation line segment of the comparison sample group with the smallest slope is taken as the iterative screening line segment of the security basic large model.)

[0110] The beneficial effects of the above technology are as follows: Based on the iterative sorting of all comparison sample pairs, the iterative screening line of the security basic large model is obtained, and the drawing process can be used to filter out the screening comparison sample pairs. A specific method for drawing the iterative screening line segment of the security basic large model is given in detail.

[0111] Example 9: Based on Example 8, the third iteration submodule of the security fundamental model based on preference reinforcement learning includes:

[0112] The iteration unit is used to treat the iteration observation point with the smallest ordinate among all iteration observation points as the marker point, and the line passing through the marker point and perpendicular to the iteration screening line of the security basic model as the iteration screening vertical line of the security basic model. The comparison sample pairs corresponding to all iteration observation points whose positions are simultaneously below the iteration screening line and the iteration screening vertical line of the security basic model are treated as the screening comparison sample pairs, and the unlabeled videos corresponding to the screening comparison sample pairs are treated as the screening iteration videos.

[0113] The modeling unit is used to take all the selected and iterated videos as training samples for the pre-trained model (that is, take all the selected and iterated videos as training input and the portrait descriptions of the people in the corresponding selected and iterated videos as training output) to obtain a large basic security model.

[0114] In this embodiment, the filtered iterative videos are a subset of unlabeled videos selected from a preset group of unlabeled videos that can be used as training samples for the pre-trained model.

[0115] The beneficial effects of the above technology are as follows: Based on the iterative filtering line of the security basic model, the filtered iterative video is obtained, and then based on the filtered iterative video, the security basic model is obtained, realizing a model that can accurately derive the correct portrait description of the input video by using a small amount of training data.

[0116] Example 10: This invention provides a method for training a large-scale security basic model based on preference reinforcement learning, which is applied to train any of the large-scale security basic models based on preference reinforcement learning in Examples 1 to 9. (Refer to...) Figure 2 ,include:

[0117] S1: Based on the pre-trained model and the input video group, obtain the portrait descriptions of the people in all videos of the input video group, and based on the portrait descriptions of the people in all videos of the input video group, obtain all positive samples and all negative samples of the input video group.

[0118] S2: Based on all positive and negative samples of the input video group, obtain the high-quality security model and the low-quality security model, and based on the high-quality security model, the low-quality security model and the preset unlabeled video group, obtain all comparison sample pairs;

[0119] S3: Based on all comparison sample pairs and the preset preference learning technique, obtain the loss value and priority iteration value of all comparison sample pairs, and based on the priority iteration value of all comparison sample pairs, obtain the iterative ranking of all comparison sample pairs;

[0120] S4: Based on the iterative sorting of all comparison sample pairs, obtain the iterative screening line of the security basic large model, and based on the iterative screening line of the security basic large model, obtain the security basic large model.

[0121] The beneficial effects of the above technology are as follows: Based on the portrait descriptions of all videos in the input video group, all positive and negative samples of the input video group are obtained, which facilitates the subsequent acquisition of high-quality and low-quality security models based on all positive and negative samples of the input video group. Based on the high-quality and low-quality security models and the preset unlabeled video group, all comparison sample pairs are obtained. Based on all comparison sample pairs and the preset preference learning technique, the loss value and priority iteration value of all comparison sample pairs are obtained. This realizes the quantification of the degree of violation of the preference relationship of the pre-trained model for each comparison sample pair and the quantification of the priority of the corresponding video of each comparison sample pair as the training iteration input of the pre-trained model. Then, based on the priority iteration value of all comparison sample pairs, the iterative ranking of all comparison sample pairs is obtained. Based on the iterative ranking of all comparison sample pairs, the iterative screening line of the basic security model is obtained. The line can be plotted and processed to select the comparison sample pairs. Finally, based on the iterative screening line of the basic security model, the basic security model is obtained. This realizes a model that can accurately derive the correct portrait description of the input video using a small amount of training data.

[0122] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from the spirit and scope of this invention, and this invention is also intended to include these modifications and variations.

Claims

1. A fundamental security model based on preference reinforcement learning, characterized in that, include: The processing module is used to obtain the portrait descriptions of people in all videos of the input video group based on the pre-trained model and the input video group, and to obtain all positive samples and all negative samples of the input video group based on the portrait descriptions of people in all videos of the input video group. The training module is used to obtain a high-quality security model and a low-quality security model based on all positive and negative samples of the input video group. Based on the high-quality security model, the low-quality security model and the preset unlabeled video group, it obtains all comparison sample pairs. The comparison sample pairs are the comparison combination of high-quality and low-quality portrait descriptions of each unlabeled video in the preset unlabeled video group. The calculation module is used to obtain the loss value and priority iteration value of all comparison sample pairs based on all comparison sample pairs and the preset preference learning technique, and to obtain the iterative ranking of all comparison sample pairs based on the priority iteration value of all comparison sample pairs. The iterative module is used to obtain the iterative screening line of the security basic model based on the iterative sorting of all comparison sample pairs, and to obtain the security basic model based on the iterative screening line of the security basic model.

2. The security foundation model based on preference reinforcement learning according to claim 1, characterized in that, The processing module includes: The first processing submodule is used to sequentially input each video of the input video group into the pre-trained model to obtain the portrait description of the person in each video of the input video group; The second processing submodule is used to obtain all positive samples and all negative samples of the input video group based on the portrait descriptions of all videos in the input video group.

3. The security foundation model based on preference reinforcement learning according to claim 2, characterized in that, The second processing submodule includes: The first processing unit is used to input the portrait description of each video in the input video group into a preset large language model to obtain the reasoning action result of each video in the input video group. The second processing unit is used to determine whether the reasoning action result of each video in the input video group is the same as the real label. If so, the corresponding video in the input video group is treated as a positive sample of the input video group; otherwise, the corresponding video in the input video group is treated as a negative sample of the input video group.

4. The security basic model based on preference reinforcement learning according to claim 1, characterized in that, The training module includes: The first training submodule is used to obtain a high-quality security model by using only all positive samples of the input video group as training samples of the pre-trained model, and to obtain a low-quality security model by using only all negative samples of the input video group as training samples of the pre-trained model. The second training submodule is used to obtain all comparison sample pairs based on the high-quality security model, the low-quality security model, and the preset unlabeled video group.

5. The security basic model based on preference reinforcement learning according to claim 4, characterized in that, The second training submodule includes: The first training unit is used to receive all unlabeled videos from a preset unlabeled video group, input each unlabeled video from the preset unlabeled video group into the high-quality security model, obtain a high-quality portrait description of each unlabeled video from the preset unlabeled video group, and input each unlabeled video from the preset unlabeled video group into the low-quality security model, obtain a low-quality portrait description of each unlabeled video from the preset unlabeled video group. The second training unit is used to treat the high-quality and low-quality portrait descriptions of each unlabeled video in the preset unlabeled video group as a comparison sample pair.

6. The security foundation model based on preference reinforcement learning according to claim 1, characterized in that, The calculation module includes: The first calculation submodule is used to obtain the high-quality description score and the low-quality description score of each comparison sample pair based on all comparison sample pairs and the preset preference learning technique, and to obtain the loss value of each comparison sample pair based on the high-quality description score and the low-quality description score of each comparison sample pair. The second calculation submodule is used to obtain the priority iteration value of each comparison sample pair based on the loss value of all comparison sample pairs, and to obtain the iterative ranking of all comparison sample pairs based on the priority iteration values ​​of all comparison sample pairs.

7. The security foundation model based on preference reinforcement learning according to claim 6, characterized in that, The second calculation submodule includes: The first calculation unit is used to obtain the priority iteration value for each comparison sample pair based on the loss value, high-quality description score, and low-quality description score of all comparison sample pairs. The second calculation unit is used to sort all comparison sample pairs in ascending order of priority iteration value to obtain the iterative sort of all comparison sample pairs.

8. The security basic model based on preference reinforcement learning according to claim 5, characterized in that, The iteration module includes: The first iteration submodule is used to take the priority iteration value of all comparison sample pairs as the x-axis value and the loss value of the corresponding comparison sample pair as the y-axis value to obtain all iteration observation points. It also takes two adjacent comparison sample pairs in the iteration sorting of all comparison sample pairs as a comparison sample pair group, and connects the iteration observation points corresponding to the two comparison sample pairs in each comparison sample pair group to obtain the iteration observation line segment of each comparison sample pair group. The second iteration submodule is used to define the ordinal numbers of the iterative observation line segments of all comparison sample pairs in ascending order of slope, starting from 1, to obtain the ordinal definition results of the iterative observation line segments of all comparison sample pairs. Based on the ordinal definition results of the iterative observation line segments of all comparison sample pairs, the iterative screening line segments of the security basic large model are obtained. The two ends of the iterative screening line segments of the security basic large model are extended to obtain the iterative screening line of the security basic large model. The third iteration submodule is used to iteratively filter straight lines based on the security basic model to obtain the security basic model.

9. The security basic model based on preference reinforcement learning according to claim 8, characterized in that, The third iteration submodule includes: The iteration unit is used to treat the iteration observation point with the smallest ordinate among all iteration observation points as the marker point, and the line passing through the marker point and perpendicular to the iteration screening line of the security basic model as the iteration screening vertical line of the security basic model. The comparison sample pairs corresponding to all iteration observation points whose positions are simultaneously below the iteration screening line and the iteration screening vertical line of the security basic model are treated as the screening comparison sample pairs, and the unlabeled videos corresponding to the screening comparison sample pairs are treated as the screening iteration videos. The modeling unit is used to use all the selected and iterated videos as training samples for the pre-trained model to obtain a large basic security model.

10. A method for training a large-scale security infrastructure model based on preference reinforcement learning, characterized in that, The method is applied to train a large-scale security basic model based on preference reinforcement learning as described in any one of claims 1 to 9, comprising: S1: Based on the pre-trained model and the input video group, obtain the portrait descriptions of the people in all videos of the input video group, and based on the portrait descriptions of the people in all videos of the input video group, obtain all positive samples and all negative samples of the input video group. S2: Based on all positive and negative samples of the input video group, obtain the high-quality security model and the low-quality security model. Based on the high-quality security model, the low-quality security model and the preset unlabeled video group, obtain all comparison sample pairs. The comparison sample pairs are the comparison combination of high-quality and low-quality portrait descriptions of each unlabeled video in the preset unlabeled video group. S3: Based on all comparison sample pairs and the preset preference learning technique, obtain the loss value and priority iteration value of all comparison sample pairs, and based on the priority iteration value of all comparison sample pairs, obtain the iterative ranking of all comparison sample pairs; S4: Based on the iterative sorting of all comparison sample pairs, obtain the iterative screening line of the security basic large model, and based on the iterative screening line of the security basic large model, obtain the security basic large model.

Images