Data processing method, electronic device and computer readable storage medium
Through a large language model, the inference of audio/video data and combined with the processing strategy of agent training, the time-consuming and labor-intensive problem of existing audio and video algorithm solutions in the construction and testing process is solved, and the audio/video algorithm solutions are improved effect and accuracy.
Patent Information
- Application Number
- CN202411249626.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-06
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2044-09-06
AI Technical Summary
Existing audio and video algorithm solutions are time-consuming and labor-intensive in the formation and testing process, and the results of large language models may not solve the actual customer problems and are not effective.
A large language model is used to infer the target sound/video data, obtain the prediction results, and train the agent to determine the matching processing strategy, and perform logical reasoning to solve the practical problems.
It reduces the workload of audio/video algorithm solutions, and improves the effectiveness of the solution, which can accurately solve customer practical problems.
Smart Images

Figure CN118761474B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, an electronic device, and a computer-readable storage medium. Background Art
[0002] At present, most algorithm solutions for target fields (such as audio and video fields) still use multiple types of algorithms in series and parallel combinations, such as target detection, target recognition, target tracking, specific rule alarms, target segmentation, target optimization, face recognition and other types of algorithms, among which these types of algorithms are further subdivided into multiple small types of algorithms. The algorithm solution for a specific scenario is the combination of these algorithms to finally obtain the target algorithm result and solve the actual problem. However, this method currently involves many types of algorithms. If there is a problem with the algorithm effect of one link, it will affect the overall effect. In addition, every time facing a new scenario problem, professional technicians are required to combine these algorithms and solve the new problem through multiple modifications and tests, which is very time-consuming and laborious.
[0003] If the large model can be applied to the algorithm solution of the target field (such as audio field, video field), the workload of building the algorithm solution can be greatly reduced. However, the results given by the large language model may not be able to solve the customer's actual problem, or may be very different from the answer the user needs. Summary of the invention
[0004] The main technical problem solved by the present application is to provide a data processing method, an electronic device and a computer-readable storage medium, which can reduce the workload of constructing an audio / video algorithm solution while improving the effect of the audio / video algorithm solution.
[0005] To solve the above technical problems, a technical solution adopted in the present application is: to provide a data processing method, the method comprising: using a large language model to infer the target audio / video data to obtain a first audio / video prediction result; and using the sample audio / video prediction result to train an intelligent agent; wherein the sample audio / video prediction result is obtained by inferring the sample audio / video data using a large language model; using the trained intelligent agent based on the first audio / video prediction result, determining a target audio / video processing strategy that matches the first audio / video prediction result; using the target audio / video processing strategy, processing the first audio / video prediction result to obtain a target audio / video processing result.
[0006] In order to solve the above technical problems, another technical solution adopted by the present application is: to provide an electronic device, the electronic device includes a memory and a processor, the memory stores program instructions, and the program instructions can be executed to implement the above data processing method.
[0007] In order to solve the above technical problems, another technical solution adopted in the present application is to provide a computer-readable storage medium, which is used to store program instructions, and the program instructions can be executed to implement the above data processing method.
[0008] In the above technical solution, the first audio / video prediction result corresponding to the target audio / video data is obtained by reasoning the target audio / video data using a large language model, that is, the audio / video algorithm solution for the target audio / video data is based on a large language model, and the large language model is used as a basic model; compared with the traditional audio / video algorithm solution for the target audio / video data (not based on a large language model), the workload of constructing an audio / video algorithm solution can be greatly reduced. In addition, the intelligent agent completes the selection of the target audio / video processing strategy that matches the first audio / video prediction result, and the selected target audio / video processing strategy can accurately perform logical reasoning on the first audio / video prediction result, so that the target audio / video processing result obtained after logical reasoning can solve the actual problem of the customer and improve the model reasoning effect. Therefore, while reducing the workload of constructing an audio / video algorithm solution, the effect of the audio / video algorithm solution can also be improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0009] Figure 1 It is a flowchart of an embodiment of a data processing method provided by the present application;
[0010] Figure 2 yes Figure 1 The flowchart of step S13 is shown as an embodiment;
[0011] Figure 3 yes Figure 2 The flowchart of step S22 of an embodiment is shown;
[0012] Figure 4 It is a structural schematic diagram of an embodiment of an electronic device provided by the present application;
[0013] Figure 5 It is a structural schematic diagram of an embodiment of a computer-readable storage medium provided by the present application. DETAILED DESCRIPTION
[0014] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.
[0015] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.
[0016] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the objects associated before and after are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of, for example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C.
[0017] See also Figure 1 , Figure 1 1 is a flow chart of an embodiment of a data processing method provided by the present application. It should be noted that if there are substantially the same results, this embodiment does not necessarily refer to the embodiment of the present invention. Figure 1 The process sequence shown is limited. Figure 1 As shown, this embodiment includes:
[0018] Step S11: Use the large language model to infer the target audio / video data to obtain a first audio / video prediction result.
[0019] In this embodiment, the target audio / video data is inferred using a large language model to obtain a first audio / video prediction result. The first audio / video prediction result corresponding to the target audio / video data is obtained by inferring the target audio / video data using a large language model, that is, the audio / video algorithm solution for the target audio / video data is based on the large language model, and the large language model is used as the basic model; compared to the traditional audio / video algorithm solution for the target audio / video data (not based on a large language model), while ensuring the accuracy of the prediction of the target audio / video data, the workload of constructing the audio / video algorithm solution can be greatly reduced. For example, traditional audio / video algorithm solutions for audio / video data mostly use multiple types of algorithms in a serial and parallel combination. However, this method currently involves many types of algorithms, and requires professional technicians to combine, modify and test them multiple times to obtain the final audio / video algorithm solution. The workload is large and very time-consuming. Applying the large language model to the audio / video algorithm solution, that is, the audio / video algorithm solution for the target audio / video data is based on the large language model, which takes the large language model as the basic model, can greatly reduce the workload of building an audio / video algorithm solution.
[0020] It should be noted that, taking a video frame in which the target audio / video data is audio data as an example, the large language model infers the video frame, and there may be multiple first video prediction results corresponding to the video frame. The multiple first video prediction results may be the categories of target objects in the video frame, the attributes of target objects, the scenes of the video frame, etc.
[0021] In one embodiment, the target audio / video data is inferred using a large language model to obtain a first audio / video prediction result, specifically: the target audio / video data and the prompt word are input into the large language model, and the large language model infers the target audio / video data based on the prompt word to obtain a first audio / video prediction result corresponding to the target audio / video data. For example, taking the target audio / video data as audio data or video data as an example: an audio frame in the audio data or a video frame in the video data, as well as the prompt word, are input into the large language model; the large language model infers the audio frame or the video frame based on the prompt word to obtain the corresponding first audio / video prediction result.
[0022] In one embodiment, the target audio / video data can be inferred using a large language model in real time to obtain a first audio / video prediction result about the target audio / video data. Of course, in other embodiments, the first audio / video prediction result about the target audio / video data can also be obtained from local storage or cloud storage, which is not limited here.
[0023] Step S12: Use the sample audio / video prediction results to train the intelligent agent.
[0024] In this implementation, the agent is trained using sample audio / video prediction results, wherein the sample audio / video prediction results are obtained by inferring the sample audio / video data using a large language model.
[0025] The first audio / video prediction result corresponding to the target audio / video data is obtained by inferring the target audio / video data using the large language model, that is, the audio / video algorithm solution for the target audio / video data is based on the large language model, and the large language model is used as the basic model. Although applying the large language model to the audio / video algorithm solution for the target audio / video data can reduce the workload of building the audio / video algorithm solution, the large language model lacks the ability to process the first audio / video prediction result it outputs, that is, the large language model lacks the ability to perform logical reasoning on the first audio / video prediction result it outputs, so that the first audio / video prediction result it outputs usually cannot solve the actual problem of the customer, and the model reasoning effect is poor.
[0026] Therefore, in this embodiment, the intelligent agent is trained using the sample audio / video prediction results to obtain an intelligent agent with converged training, and the intelligent agent can subsequently be used to determine the target audio / video processing strategy that matches the first audio / video prediction result based on the first audio / video prediction result. The intelligent agent is introduced, and the intelligent agent completes the selection of the target audio / video processing strategy that matches the first audio / video prediction result. The selected target audio / video processing strategy can accurately perform logical reasoning on the first audio / video prediction result, so that the result obtained after logical reasoning can solve the actual problem of the customer.
[0027] The introduction of the intelligent agent can be regarded as a combination of a large language model and an intelligent agent. The large language model completes the model reasoning part (reasoning the target audio / video data to obtain the first audio / video prediction result), and the intelligent agent completes the task of selecting the target audio / video processing strategy. It can be adaptively applied to multiple scenarios, reducing the repetitive workload caused by scene changes, reducing the number of models, and improving the model reasoning effect. In other words, the large language model is used to perform model reasoning on the target audio / video data; the target audio / video processing strategy determined by the intelligent agent is used to perform logical reasoning on the first audio / video prediction result obtained by the large language model reasoning on the target audio / video data; then, it can be regarded as decoupling the model reasoning part and the logical reasoning part in the traditional audio / video algorithm solution for the target audio / video data, where the model reasoning part is completed by the large language model, and the logical reasoning part is completed by the target audio / video processing strategy determined by the intelligent agent.
[0028] Step S13: Determine a target audio / video processing strategy that matches the first audio / video prediction result based on the first audio / video prediction result using the trained intelligent agent.
[0029] In this implementation, a trained intelligent agent is used to determine a target audio / video processing strategy that matches the first audio / video prediction result based on the first audio / video prediction result. An intelligent agent is introduced to select a target audio / video processing strategy that matches the first audio / video prediction result. The selected target audio / video processing strategy can accurately perform logical reasoning on the first audio / video prediction result, so that the result obtained after logical reasoning can solve the actual problem of the customer.
[0030] The introduction of the intelligent agent can be regarded as a combination of a large language model and an intelligent agent. The large language model completes the model reasoning part (reasoning the target audio / video data to obtain the first audio / video prediction result), and the intelligent agent completes the task of selecting the target audio / video processing strategy. It can be adaptively applied to multiple scenarios, reducing the repetitive workload caused by scene changes, reducing the number of models, and improving the model reasoning effect. In other words, the large language model is used to perform model reasoning on the target audio / video data; the target audio / video processing strategy determined by the intelligent agent is used to perform logical reasoning on the first audio / video prediction result obtained by the large language model reasoning on the target audio / video data; then, it can be regarded as decoupling the model reasoning part and the logical reasoning part in the traditional audio / video algorithm solution for the target audio / video data, where the model reasoning part is completed by the large language model, and the logical reasoning part is completed by the target audio / video processing strategy determined by the intelligent agent.
[0031] There are multiple first audio / video prediction results. In one embodiment, the target audio / video processing strategy that matches the first audio / video prediction result can be determined directly based on all the first audio / video prediction results. Of course, in other embodiments, the first audio / video prediction result required for determining the target audio / video processing strategy can also be screened out from multiple first audio / video prediction results, and then the target audio / video processing strategy is determined based on the first audio / video prediction result required for determining the target audio / video processing strategy. On the one hand, the first audio / video prediction results that need to be paid attention to are reduced, which improves the efficiency of determining the target audio / video processing strategy; on the other hand, only the first audio / video prediction results required for determining the target audio / video processing strategy are retained, avoiding the existence of useless first audio / video prediction results for determining the target audio / video processing strategy. The accuracy of the determined target audio / video processing strategy is affected, thereby avoiding the accuracy of the target audio / video processing result obtained based on the target audio / video processing strategy being affected by the inaccurate determined target audio / video processing strategy.
[0032] In one embodiment, the target audio / video processing strategy includes a target audio / video processing algorithm combination. The target audio / video processing algorithm combination that matches the first audio / video prediction result is determined using a trained intelligent agent, and the determined target audio / video processing algorithm combination can accurately perform logical reasoning on the first audio / video prediction result, so that the target audio / video processing result obtained after logical reasoning can accurately solve the actual problem of the customer and improve the effect of model reasoning.
[0033] Step S14: using the target audio / video processing strategy, processing the first audio / video prediction result to obtain the target audio / video processing result.
[0034] In this embodiment, the first audio / video prediction result is processed using the target audio / video processing strategy to obtain the target audio / video processing result. Since the target audio / video processing strategy is an audio / video processing strategy that matches the first audio / video prediction result and is determined by the intelligent agent, the first audio / video prediction result is processed using the target audio / video processing strategy that matches the first audio / video prediction result, and the first audio / video prediction result can be accurately logically inferred, so that the target audio / video processing result obtained by logical inference can solve the actual problem of the customer, thereby improving the model reasoning effect.
[0035] The first audio / video prediction result corresponding to the target audio / video data is obtained by reasoning the target audio / video data using a large language model, that is, the algorithm solution for the target audio / video data is based on a large language model, and the large language model is used as the basic model; compared with the traditional audio / video algorithm solution for the target audio / video data (not based on a large language model), the workload of constructing an audio / video algorithm solution can be greatly reduced. In addition, the intelligent agent completes the selection of the target audio / video processing strategy that matches the first audio / video prediction result, and the selected target audio / video processing strategy can accurately perform logical reasoning on the first audio / video prediction result, so that the target audio / video processing result obtained after logical reasoning can solve the actual problem of the customer and improve the model reasoning effect. Therefore, while reducing the workload of constructing an audio / video algorithm solution, the effect of the audio / video algorithm solution can also be improved.
[0036] Among them, the target audio / video processing results include rule alarm results, tracking results, optimization results, etc., which are not limited here.
[0037] See also Figure 2 , Figure 2 yes Figure 1 It should be noted that if there is substantially the same result, this embodiment does not use Figure 2 The process sequence shown is limited. Figure 2 As shown, there are multiple first audio / video prediction results, and this embodiment includes:
[0038] Step S21: Filter out at least one second audio / video prediction result from multiple first audio / video prediction results.
[0039] In this embodiment, at least one second audio / video prediction result is screened out from a plurality of first audio / video prediction results, wherein the second audio / video prediction result is the first audio / video prediction result required for determining the target audio / video processing strategy. That is to say, the first audio / video prediction result required for determining the target audio / video processing strategy is screened out from a plurality of first audio / video prediction results, and the target audio / video processing strategy can be subsequently determined based on the screened first audio / video prediction result. On the one hand, the number of first audio / video prediction results that need to be paid attention to is reduced, thereby improving the efficiency of determining the target audio / video processing strategy; on the other hand, only the first audio / video prediction result required for determining the target audio / video processing strategy is retained, thereby avoiding the accuracy of the determined target audio / video processing strategy being affected by the existence of the first audio / video prediction result that is useless for determining the target audio / video processing strategy, thereby avoiding the accuracy of the target audio / video processing result obtained based on the target audio / video processing strategy being affected by the inaccurate determined target audio / video processing strategy.
[0040] For example, a large language model is used to infer the target audio / video data to obtain a first audio / video prediction result, including a first audio / video prediction result A and a first audio / video prediction result B; since the first audio / video prediction result A is required to determine the target audio / video processing strategy, the first audio / video prediction result A is used as the second audio / video prediction result.
[0041] For another example, a large language model is used to infer the target audio / video data to obtain a first audio / video prediction result, including a first audio / video prediction result A, a first audio / video prediction result B, and a first audio / video prediction result C; since the first audio / video prediction result A and the first audio / video prediction result B are required to determine the target audio / video processing strategy, the first audio / video prediction result A and the first audio / video prediction result B are used as the second audio / video prediction result.
[0042] In one embodiment, the target audio / video processing result is the audio / video processing result of the target object in the target audio / video data, and the second audio / video prediction result includes the association information associated with the target object and the audio / video processing rule for the target object adopted by the target audio / video data. That is to say, when performing logical reasoning on the target object, the required audio / video prediction result includes the association information associated with the target object and the audio / video processing rule for the target object adopted by the target audio / video data. The relationship information associated with the target object can reflect the relevant information of the target object; the audio / video processing rule for the target object adopted by the target audio / video data can reflect the logical reasoning required for the target object.
[0043] Among them, the associated information associated with the target object can be the category of the target object, the attributes of the target object (such as size, color, whether wearing glasses, whether there is a tail, etc.), etc.; the audio / video processing rules for the target object adopted by the target audio / video data can be rule alarm, optimization, tracking, etc.
[0044] For example, take the target audio / video data as a video frame in the video data as an example: use the large language model to infer the video frame to obtain a first video prediction result, the first video prediction result includes the category of the target object in the video frame, the attributes of the target object (including color and size), and the video processing rules used by the video frame for the target object as a rule alarm; because the first video prediction result required to determine the target video processing strategy is the category of the target object in the video frame and the video processing rules used by the video frame for the target object as a rule alarm, the category of the target object in the video frame and the video processing rules used by the video frame for the target object are used as the second video prediction result.
[0045] Step S22: Determine a target audio / video processing strategy that matches at least one second audio / video prediction result based on at least one second audio / video prediction result using a trained agent.
[0046] In this embodiment, a trained intelligent agent is used to determine a target audio / video processing strategy that matches at least one second audio / video prediction result based on at least one second audio / video prediction result. The at least one second audio / video prediction result is a first audio / video prediction result required for determining the target audio / video processing strategy selected from multiple first audio / video prediction results. Therefore, the use of a trained intelligent agent to determine a target audio / video processing strategy that matches at least one second audio / video prediction result based on at least one second audio / video prediction result is to use an intelligent agent to determine the target audio / video processing strategy based on the first audio / video prediction result required for determining the target audio / video processing strategy. On the one hand, the number of first audio / video prediction results that need to be paid attention to is reduced, thereby improving the efficiency of determining the target audio / video processing strategy; on the other hand, only the first audio / video prediction result required for determining the target audio / video processing strategy is retained, thereby avoiding the accuracy of the determined target audio / video processing strategy being affected by the existence of the first audio / video prediction result that is useless for determining the target audio / video processing strategy, thereby avoiding the accuracy of the target audio / video processing result obtained based on the target audio / video processing strategy being affected by the inaccurate determined target audio / video processing strategy.
[0047] Furthermore, at this time, the target audio / video processing strategy is used to process the first audio / video prediction result to obtain the target audio / video processing result, specifically: the target audio / video processing strategy is used to process at least one second audio / video prediction result to obtain the target audio / video processing result. In other words, the target audio / video processing strategy is used to perform logical reasoning more specifically, and the obtained logical reasoning result is more accurate.
[0048] For example, take the target audio / video data as a video frame in the video data as an example: use the large language model to infer the video frame to obtain a first video prediction result, the first video prediction result includes the category of the target object in the video frame, the attributes of the target object (including color and size), and the video processing rules used by the video frame for the target object as a rule alarm; because the first video prediction result required to determine the target video processing strategy is the category of the target object in the video frame and the video processing rules used by the video frame for the target object as a rule alarm, the category of the target object in the video frame and the video processing rules used by the video frame for the target object are used as the second video prediction result.
[0049] Furthermore, the intelligent agent is used to issue a rule alarm based on the category of the target object in the video frame and the video processing rules regarding the target object adopted in the video frame, and a matching target video processing strategy is determined; then, the matching target video processing strategy is used to perform processing based on the category of the target object in the video frame and the video processing rules regarding the target object adopted in the video frame, and a rule alarm result is obtained to determine whether to perform alarm processing.
[0050] In one embodiment, if Figure 3 As shown, Figure 3 yes Figure 2 The flowchart of step S22 of an embodiment shown in FIG. 1 is a flowchart of determining a target audio / video processing strategy that matches at least one second audio / video prediction result based on at least one second audio / video prediction result using a trained agent, specifically including the following sub-steps:
[0051] Step S31: Use the trained intelligent agent to extract features from the second audio / video prediction result to obtain target features.
[0052] In this embodiment, the trained intelligent agent is used to extract features of the second audio / video prediction result to obtain target features. In other words, the intelligent agent is used to extract abstract features of the second audio / video prediction result.
[0053] In one embodiment, the intelligent agent includes a second intelligent agent, and feature extraction is performed on the second audio / video prediction result to obtain the target feature by using the second intelligent agent in the intelligent agent.
[0054] In a specific embodiment, the second agent is composed of at least two different neural networks, and at least one of the following network parameters of the at least two different neural networks is different: the number of layers of the neural network, the number of neurons, and the type of the neural network. In other words, the second agent in the agent is composed of a variety of different neural networks, wherein the main differences are the number of layers of the neural network, the number of neurons, the type of the neural network, etc.
[0055] In one embodiment, the training steps of the second agent include: step 1: obtaining sample audio / video prediction results, wherein the sample audio / video prediction results are marked with real audio / video processing strategies; step 2: using the second agent to extract features of the sample audio / video prediction results to obtain sample features; step 3: based on the sample features, predicting the sample audio / video processing strategies and corresponding weights; step 4: adjusting the parameters of the second agent based on the real audio / video processing strategies, the sample audio / video processing strategies and the corresponding weights. The second agent is trained using the sample audio / video prediction results so that the second agent learns the mapping relationship between the audio / video prediction results and the audio / video processing strategies, and can accurately determine the matching audio / video processing strategies when it is subsequently given an audio / video prediction result. It should be noted that, for the second agent whose training has converged, its final output audio / video processing strategy is not used as the input of the first agent, but its extracted features are used as the input of the first agent; that is, in the process of the agent determining the target audio / video processing strategy that matches at least one second audio / video prediction result based on at least one second audio / video prediction result, the second agent of the agent provides the ability to extract abstract features.
[0056] The loss function used in the training of the second agent is as follows:
[0057]
[0058] Where F represents the loss function; m and n represent the number of sample audio / video processing strategy categories and sample size, respectively; Indicates the label that the i-th sample belongs to the sample audio / video processing strategy category j (real audio / video processing strategy), Represents the probability that sample i is predicted to be sample audio / video processing strategy category j.
[0059] In a specific implementation, the sample audio / video prediction results are obtained by establishing target types, attribute warehouses, and rule warehouses, and by running different scenarios and different audio / video algorithm solutions. The sample audio / video processing strategy is used as a sample logic reasoning algorithm combination as an example to provide a detailed description of constructing diverse data.
[0060] Specifically, (1) obtaining audio / video algorithm solutions for different scenarios and different intelligent services, and obtaining the categories that can be detected in the audio / video algorithm solutions for different scenarios and different intelligent services, as shown below:
[0061]
[0062] Among them, S1 represents the set of categories that can be detected by the audio / video algorithm solutions of different intelligent services in different scenarios, label1 represents one of the target categories; and n represents the total number of categories supported for detection.
[0063] (2) Obtaining audio / video algorithm solutions for different scenarios and different intelligent services can identify the attributes corresponding to the above target categories, as shown below:
[0064]
[0065] Among them, Slabeli represents the attribute set corresponding to the i-th target category; attr1 represents one of the attributes in the i-th target category; m represents the total number of attributes supported for recognition.
[0066] (3) Obtain the rules that can be supported in the audio / video algorithm solutions for different scenarios and different intelligent services, as shown below:
[0067]
[0068] Among them, Srule represents all rule sets supported by audio / video algorithm solutions for different scenarios and different intelligent services; rule1 represents one of the supported rules; and k represents the total number of supported rules.
[0069] (4) Obtain all scenarios supported by audio / video algorithm solutions for different scenarios and different intelligent services, as shown below:
[0070]
[0071] Among them, Sscene represents the set of all scenes supported by the audio / video algorithm solutions of different intelligent services in different scenes, scene1 represents one of the supported scenes; j represents the total number of supported scenes;
[0072] (5) Obtain the logical reasoning part (such as tracking, rule alarm, optimization, etc.) of the audio / video algorithm solutions for different scenarios and different intelligent services, as shown below:
[0073]
[0074] Among them, Sscene represents all audio / video algorithm solutions supported in a scene; solu1 represents the permutation and combination pipeline of the logical reasoning algorithm, that is, the logical reasoning part; h represents the total number of audio / video algorithm solutions supported.
[0075] (6) Obtain all audio / video algorithm solutions supported in the scenario, run these audio / video algorithm solutions, select rules from all rule sets supported by audio / video algorithm solutions for different scenarios and different intelligent services and configure them as scenario rules; further, obtain the result data of the audio / video algorithm solutions for different scenarios and different intelligent services running in different scenarios, as shown below:
[0076]
[0077] Among them, data(g,f) represents the f-th frame image result set output by the g-th audio / video algorithm solution, label represents the target category, attr represents the attribute combination corresponding to the target category, and alert represents the processing rule used for the current frame of image data.
[0078] (7) Obtain a data set for training the agent, as shown below:
[0079]
[0080] Among them, Sdata represents the data set used for training the intelligent agent; (data(g,f);pipe)1 represents the first piece of data; a represents the total number of data; pipe represents the combination of logical reasoning algorithms used for these data.
[0081] Step S32: Based on the target features, predict at least two reference audio / video processing strategies.
[0082] In this embodiment, at least two reference audio / video processing strategies are predicted based on the target features. That is, at least two reference audio / video processing strategies are predicted based on the target features, providing multiple reference audio / video processing strategies for selection, that is, providing selection diversity for subsequent selection.
[0083] In one embodiment, the agent includes a first agent, and the first agent includes at least two different prediction modules, each of which is used to predict a reference audio / video processing strategy based on the target feature. That is, the first agent includes at least two different prediction modules, each of which predicts based on the target feature to obtain a corresponding reference audio / video processing strategy.
[0084] Among them, the prediction module can be a linear regression module, a decision tree module, a random forest module, a support vector machine module, etc., which is not limited here.
[0085] Step S33: Filter out a target audio / video processing strategy from at least two reference audio / video processing strategies.
[0086] In this embodiment, the target audio / video processing strategy is selected from at least two reference audio / video processing strategies. In one embodiment, the intelligent agent further includes a third intelligent agent, and the target audio / video processing strategy is selected from at least two reference audio / video processing strategies by the third intelligent agent. The third layer of intelligent agents provides stability to the intelligent agents and reduces the misjudgment rate.
[0087] In one embodiment, a target audio / video processing strategy is selected from at least two reference audio / video processing strategies, specifically by: obtaining target weights corresponding to reference audio / video processing strategies of different categories; and selecting the target audio / video processing strategy from the reference audio / video processing strategies of each category based on the target weights corresponding to the reference audio / video processing strategies of each category. In other words, the target audio / video processing strategy is determined based on the weights of the reference audio / video processing strategies of the same category.
[0088] In a specific implementation, the target weights corresponding to the reference audio / video processing strategies of different categories are obtained, specifically: for each category of reference audio / video processing strategy, the reference weights corresponding to each reference audio / video processing strategy belonging to the category are obtained; the sum of the reference weights corresponding to each reference audio / video processing strategy of the category is used as the target weight corresponding to the reference audio / video processing strategy of the category. For example, it includes reference audio / video processing strategy A and the corresponding reference weight 0.3, reference audio / video processing strategy B and the corresponding reference weight 0.1, reference audio / video processing strategy C and the corresponding reference weight 0.2, reference audio / video processing strategy A and the corresponding reference weight 0.2 and reference audio / video processing strategy D and the corresponding reference weight 0.2; the target weight corresponding to reference audio / video processing strategy A is 0.5, and the target weights of other reference audio / video processing strategies are their own reference weights.
[0089] In a specific implementation, based on the target weights corresponding to the reference audio / video processing strategies of each category, the target audio / video processing strategy is screened out from the reference audio / video processing strategies of each category, specifically: the reference audio / video processing strategy of the category corresponding to the maximum target weight is used as the target audio / video processing strategy. For example, it includes reference audio / video processing strategy A and the corresponding reference weight 0.3, reference audio / video processing strategy B and the corresponding reference weight 0.1, reference audio / video processing strategy C and the corresponding reference weight 0.2, reference audio / video processing strategy A and the corresponding reference weight 0.2 and reference audio / video processing strategy D and the corresponding reference weight 0.2; the target weight corresponding to reference audio / video processing strategy A is 0.5, and the target weights of other reference audio / video processing strategies are their own reference weights; then, reference audio / video processing strategy A is used as the target audio / video processing strategy.
[0090] See also Figure 4 , Figure 4 4 is a schematic diagram of the structure of an embodiment of an electronic device provided by the present application. The electronic device 40 includes a memory 41 and a processor 42 coupled to each other, and the processor 42 is used to execute program instructions stored in the memory 41 to implement the steps of any of the above data processing method embodiments. In a specific implementation scenario, the electronic device 40 may include but is not limited to: a microcomputer, a server, and in addition, the electronic device 40 may also include a mobile device such as a laptop computer and a tablet computer, which is not limited here.
[0091] Specifically, the processor 42 is used to control itself and the memory 41 to implement the steps of any of the above data processing method embodiments. The processor 42 can also be called a CPU (Central Processing Unit). The processor 42 may be an integrated circuit chip with signal processing capabilities. The processor 42 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 42 can be implemented by an integrated circuit chip.
[0092] See also Figure 5 , Figure 5It is a structural diagram of an embodiment of a computer-readable storage medium provided by the present application. The computer-readable storage medium 50 of the embodiment of the present application stores a program instruction 51, and when the program instruction 51 is executed, it implements the method provided by any embodiment of the data processing method of the present application and any non-conflicting combination. Among them, the program instruction 51 can form a program file and be stored in the above-mentioned computer-readable storage medium 50 in the form of a software product, so that a computer device (which can be a personal computer, a server, or a network device, etc.) executes all or part of the steps of the methods of each implementation method of the present application. The aforementioned computer-readable storage medium 50 includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk, or a terminal device such as a computer, a server, a mobile phone, and a tablet.
[0093] If the technical solution of this application involves personal information, the product using the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to the collection of his or her personal information; or on the device that processes personal information, the personal information processing rules are notified by obvious signs / information, and the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.
[0094] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.
Claims
1. A data processing method, characterized in that: The method comprises: Using the large language model to infer the target audio / video data to obtain a first audio / video prediction result; and, The agent is trained using sample audio / video prediction results; wherein the sample audio / video prediction results are obtained by reasoning the sample audio / video data using a large language model; Determine, using the trained agent based on the first audio / video prediction result, a target audio / video processing strategy that matches the first audio / video prediction result; Using the target audio / video processing strategy, processing the first audio / video prediction result to obtain a target audio / video processing result; There are multiple first audio / video prediction results; using the trained agent to determine a target audio / video processing strategy that matches the first audio / video prediction result based on the first audio / video prediction result includes: Filtering out at least one second audio / video prediction result from the plurality of first audio / video prediction results; wherein the second audio / video prediction result is the first audio / video prediction result required for determining the target audio / video processing strategy; Determining, using the trained agent based on the at least one second audio / video prediction result, a target audio / video processing strategy that matches the at least one second audio / video prediction result; The using the target audio / video processing strategy to process the first audio / video prediction result to obtain the target audio / video processing result includes: The at least one second audio / video prediction result is processed using the target audio / video processing strategy to obtain the target audio / video processing result.
2. The method according to claim 1, characterized in that: The target audio / video processing result is the audio / video processing result about the target object in the target audio / video data, and the second audio / video prediction result includes association information associated with the target object and the audio / video processing rule about the target object adopted by the target audio / video data.
3. The method according to claim 1, characterized in that: The step of using the trained agent to determine a target audio / video processing strategy that matches the at least one second audio / video prediction result based on the at least one second audio / video prediction result includes: Using the trained agent to extract features from the second audio / video prediction result to obtain target features; Based on the target feature, predict at least two reference audio / video processing strategies; The target audio / video processing strategy is selected from the at least two reference audio / video processing strategies.
4. The method according to claim 3, characterized in that The step of selecting the target audio / video processing strategy from the at least two reference audio / video processing strategies includes: Obtain target weights corresponding to reference audio / video processing strategies of different categories; Based on the target weights corresponding to the reference audio / video processing strategies of each category, the target audio / video processing strategy is screened out from the reference audio / video processing strategies of each category.
5. The method according to claim 4, characterized in that The obtaining of target weights corresponding to reference audio / video processing strategies of different categories includes: For each category of reference audio / video processing strategies, obtaining a reference weight corresponding to each reference audio / video processing strategy belonging to the category; The sum of the reference weights corresponding to the reference audio / video processing strategies of the category is used as the target weight corresponding to the reference audio / video processing strategies of the category; And / or, the target audio / video processing strategy is selected from the reference audio / video processing strategies of each category based on the target weights corresponding to the reference audio / video processing strategies of each category, including: The reference audio / video processing strategy of the category corresponding to the maximum target weight is used as the target audio / video processing strategy.
6. The method according to claim 3, characterized in that The trained agent includes a first agent, which includes at least two different prediction modules, each of which is used to predict a reference audio / video processing strategy based on the target feature.
7. The method according to claim 3, characterized in that The trained intelligent agent includes a second intelligent agent and a third intelligent agent. The feature extraction of the second audio / video prediction result to obtain the target feature is performed using the second intelligent agent, and the screening of the target audio / video processing strategy from the at least two reference audio / video processing strategies is performed using the third intelligent agent.
8. The method according to claim 7, characterized in that The second intelligent agent is composed of at least two different neural networks, and at least one of the following network parameters of the at least two different neural networks is different: the number of layers of the neural network, the number of neurons, and the type of the neural network.
9. The method according to claim 7, characterized in that: The training steps of the second agent include: Obtaining the sample audio / video prediction result; wherein the sample audio / video prediction result is marked with a real audio / video processing strategy; Using the second agent to extract features from the sample audio / video prediction results to obtain sample features; Based on the sample features, predict the sample audio / video processing strategy and the corresponding weight; Based on the real audio / video processing strategy, the sample audio / video processing strategy and the corresponding weights, the parameters of the second agent are adjusted; wherein the features extracted from the trained second agent are used as the input of the first agent among the agents.
10. The method according to claim 1, characterized in that The target audio / video processing result includes one of a rule alarm result, a tracking result, and a preferred result.
11. The method according to claim 1, characterized in that: The method further comprises: Obtain several audio / video algorithm solutions; Using the several audio / video algorithm solutions, inferring the reference audio / video data to obtain a reference audio / video prediction result; and, extracting a reference audio / video processing strategy from the plurality of audio / video algorithm solutions; A mapping relationship between the reference audio / video processing strategy and the reference audio / video prediction result is established.
12. An electronic device, characterized in that: The electronic device comprises a memory and a processor, wherein the memory stores program instructions, and the program instructions can be executed to implement the method according to any one of claims 1-11.
13. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program instructions, and the program instructions can be executed to implement the method according to any one of claims 1-11.
Citation Information
Patent Citations
Decision-making method, device and equipment based on large language model and reinforcement learning
CN118260681A
