Large model algorithm cold start cue word optimization method, device and equipment

Abstract

The invention provides a large model algorithm cold start cue word optimization method, device and equipment. In one example, the method comprises the steps of obtaining an initial prompt word of a target business scene; according to the initial cue word, executing a visual field task of a target business scene by using a multi-modal large model which is not pre-trained for the target business scene to obtain an output event; and under the condition of determining that a cue word optimization condition is met, according to the initial cue word, an event picture which corresponds to the output event and carries annotation information, and algorithm capability description information corresponding to the target business scene, performing cue word optimization by utilizing the multi-modal large model to obtain an optimized cue word. According to the method, automatic optimization of the cue word can be realized, and technical support is provided for improving the performance of executing a visual field task of a target business scene by using a multi-modal large model.

Description

Technical Field

[0001] The present application relates to the field of AI intelligent algorithm technology, and in particular to a method, device and equipment for optimizing cold start prompt words for a large model algorithm. Background Art

[0002] A multimodal big model (referred to as a big model in this article) refers to a large deep learning model that can simultaneously process and understand multiple types of data (such as text, images, audio, etc.).

[0003] In order to enable the big model to accurately understand business scenarios and actual needs in the visual field, prompt words are proposed in the application of big models. Based on appropriate prompt words, the performance of big models in processing visual tasks can be improved.

[0004] How to obtain appropriate prompt words has become a technical problem that needs to be solved urgently in large-model vision task processing. Summary of the invention

[0005] In view of this, the present application provides a method, device and equipment for determining an analysis area and performing intelligent analysis.

[0006] According to a first aspect of an embodiment of the present application, a method for optimizing cold start prompt words of a large model algorithm is provided, comprising:

[0007] Get the initial prompt words of the target business scenario;

[0008] According to the initial prompt word, using a multimodal large model that is not pre-trained for the target business scenario, executing a visual domain task of the target business scenario to obtain an output event;

[0009] When it is determined that the prompt word optimization conditions are met, the prompt word is optimized using the multimodal large model based on the initial prompt word, the event picture with annotation information corresponding to the output event, and the algorithm capability description information corresponding to the target business scenario to obtain the optimized prompt word; wherein the annotation information includes true positive or false positive.

[0010] According to a second aspect of an embodiment of the present application, a large model algorithm cold start prompt word optimization device is provided, comprising:

[0011] An acquisition unit, configured to acquire an initial prompt word of a target business scenario;

[0012] An execution unit is configured to execute the visual domain task of the target business scenario based on the initial prompt word and use a multimodal large model that is not pre-trained for the target business scenario to obtain an output event;

[0013] The optimization unit is configured to optimize the prompt words using the multimodal large model based on the initial prompt words, the event picture with annotation information corresponding to the output event, and the algorithm capability description information corresponding to the target business scenario to obtain the optimized prompt words when it is determined that the prompt word optimization conditions are met.

[0014] According to a third aspect of an embodiment of the present application, there is provided an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0015] Memory, used to store computer programs;

[0016] The processor is used to implement the method provided in the first aspect when executing the program stored in the memory.

[0017] According to a fourth aspect of an embodiment of the present application, a non-temporary computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the method provided in the first aspect is implemented.

[0018] The large model algorithm cold start prompt word optimization method of the embodiment of the present application obtains the initial prompt word of the target business scenario, and based on the obtained initial prompt word, uses the multimodal large model to execute the visual field task of the target business scenario to obtain the output event. Based on the initial prompt word, a positive learning and optimization initial direction is provided for the multimodal large model. There is no need to pre-train the multimodal large model for the target business scenario. When the multimodal large model algorithm is cold started, the large model can self-learn the prompt word. Then, when it is determined that the prompt word optimization conditions are met, the initial prompt word, output event, and the output event are used as the input. The event picture with annotated information corresponding to the event and the algorithm capability description information corresponding to the target business scenario are used to optimize the prompt words using the multimodal big model to obtain the optimized prompt words, and the initial prompt words are optimized using the self-learning ability of the multimodal big model. Since the multimodal big model can learn information in multiple modes such as text and pictures, prompt words that are more suitable for the target business scenario can be optimized through the multimodal big model, realizing automatic optimization of prompt words, and providing technical support for improving the performance of visual field tasks in target business scenarios using multimodal big models. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a flow chart of a large model algorithm cold start prompt word optimization method provided in an embodiment of the present application;

[0020] Figure 2It is a flow chart of another large model algorithm cold start prompt word optimization method provided in an embodiment of the present application;

[0021] Figure 3 It is a structural schematic diagram of a large model algorithm cold start prompt word optimization device provided in an embodiment of the present application;

[0022] Figure 4 It is a structural schematic diagram of another large model algorithm cold start prompt word optimization device provided in an embodiment of the present application;

[0023] Figure 5 It is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the technical solutions provided by the embodiments of the present application and to make the above-mentioned purposes, features and advantages of the embodiments of the present application more obvious and understandable, the technical solutions in the embodiments of the present application are further described in detail below in conjunction with the accompanying drawings.

[0025] It should be noted that the serial numbers of the steps in the embodiments of the present application do not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0026] See also Figure 1 , is a flow chart of a large model algorithm cold start prompt word optimization method provided in an embodiment of the present application, such as Figure 1 As shown, the large model algorithm cold start prompt word method may include the following steps:

[0027] Step S100: Obtain an initial prompt word for a target business scenario.

[0028] In the embodiments of the present application, the target business scenario does not specifically refer to a fixed business scenario, but can refer to any business scenario, for example, a fishing detection scenario, a civilized dog walking detection scenario, etc.

[0029] Exemplarily, the initial prompt word of the target business scenario can be obtained based on the basic analysis and summary of the target business scenario by relevant personnel, and is used to prompt the multimodal large model to detect the target corresponding to the prompt word.

[0030] For example, assuming that the target business scenario is a fishing detection scenario, the initial prompt words may include people, fishing rods, and waterside.

[0031] Step S110: Based on the initial prompt word, a multimodal large model that is not pre-trained for the target business scenario is used to execute the visual domain task of the target business scenario to obtain an output event.

[0032] In an embodiment of the present application, when the initial prompt words of the target business scenario are obtained, a multimodal large model, such as a CLIP (Contrastive Language-Image Pretraining) model, can be used to perform visual domain tasks of the target business scenario based on the obtained initial prompt words to obtain output events.

[0033] Exemplarily, in the process of using a multimodal large model to perform visual domain tasks in a target business scenario, for an input image to be detected, such as a screenshot of a front-end monitoring device, the multimodal large model can be used to determine whether a specified event in the target business scenario exists in the image to be detected based on the above initial prompt words. If so, the corresponding event is output. For example, for a fishing detection scenario, the above specified event can be a fishing event. When a fishing event is detected, a fishing event output can be performed to obtain an output event for the fishing detection scenario.

[0034] Exemplarily, the event image corresponding to the output event is the input image (image to be detected) in which the specified event in the target business scenario is detected.

[0035] The initial prompt word may be a text description related to a specific target in a designated event.

[0036] Among them, in the process of using the multimodal big model to perform visual field tasks based on the initial prompt words, there is no need to collect training materials in advance for the target business scenario to pre-train the multimodal big model, that is, when the multimodal big model algorithm is cold started, the learning of the initial prompt words of the multimodal big model is realized.

[0037] Exemplarily, in order to improve the detection rate of performing visual domain tasks based on initial prompt words using a multimodal large model and reduce the occurrence of missed reports, a lower initial confidence level can be set for the initial prompt words.

[0038] For example, for the initial prompt word, its initial confidence can be set to be less than a preset confidence threshold.

[0039] Exemplarily, the preset confidence threshold may be determined based on empirical values.

[0040] Among them, the confidence of the prompt word is positively correlated with the accuracy of detecting the target corresponding to the prompt word from the image using the multimodal large model.

[0041] For example, assuming that the initial prompt word includes a fishing rod and the initial confidence of the initial prompt word is 30%, then in the process of detecting the input image using the multimodal large model, if an object is detected from the image, and the probability that the multimodal large model determines that the object is a fishing rod is greater than 30%, it can be determined that a fishing rod is detected from the image.

[0042] Therefore, by setting the confidence of the initial prompt word relatively low, the detection rate of the multimodal large model for the specified event can be improved.

[0043] Step S120: When it is determined that the prompt word optimization conditions are met, the prompt word is optimized using a multimodal large model based on the initial prompt word, the event image with annotation information corresponding to the output event, and the algorithm capability description information corresponding to the target business scenario to obtain the optimized prompt word.

[0044] In an embodiment of the present application, in order to improve the performance of visual domain tasks in target business scenarios using a large multimodal model, the initial prompt words may be optimized if the prompt word optimization conditions are met.

[0045] In one example, satisfying the prompt word optimization condition may include: the number of output events reaches a preset number.

[0046] Exemplarily, when the number of output events obtained by executing the visual domain task of the target business scenario using the multimodal large model based on the initial prompt word reaches a preset number (which can be set according to the actual scenario), it is determined that the prompt word needs to be optimized.

[0047] It should be noted that, in the embodiment of the present application, the above-mentioned determination of whether to perform prompt word optimization based on the number of output events is merely a specific example of determining whether prompt word optimization is needed in the embodiment of the present application, and is not a limitation on the scope of protection of the present application. For example, it is also possible to determine that prompt word optimization is needed when the duration of the visual field task of the target business scenario performed using a multimodal large model based on the initial prompt word reaches a preset duration.

[0048] In another example, satisfying the prompt word optimization condition may include: the duration of executing the visual domain task of the target business scenario using the multimodal large model reaches a preset duration.

[0049] In another example, satisfying the prompt word optimization condition may include: the duration of executing the visual domain task of the target business scenario using the multimodal large model reaches a preset duration, and the number of output events reaches a preset number.

[0050] Accordingly, when it is determined that the prompt word optimization conditions are met, the prompt word can be optimized using a multimodal large model based on the initial prompt word, the event image with annotation information corresponding to the output event, and the algorithm capability description information corresponding to the target business scenario to obtain the optimized prompt word.

[0051] The annotation information corresponding to the event image can be determined by manual annotation, or automatically annotated by an automatic annotation model.

[0052] Exemplarily, the annotation information corresponding to the event picture may include a true positive (ie, the specified event exists in the event picture) or a false positive (ie, the specified event does not exist in the event picture).

[0053] Exemplarily, the initial prompt words, the event picture with annotation information corresponding to the output event, and the algorithm capability description information corresponding to the target business scenario can be input as prompt words into the multimodal big model, and the multimodal big model can be used to generate optimized prompt words.

[0054] Exemplarily, the algorithm capability description information corresponding to the target business scenario may include a requirement description of an algorithm that is expected to be used to execute a visual domain task of the target business scenario, which may reflect the scenario characteristics of the target business scenario.

[0055] In an example, the algorithm capability description information corresponding to the target business scenario may include algorithm name information.

[0056] For example, taking the fishing detection scenario as an example, the corresponding algorithm capability description information may include "fishing detection algorithm" (ie, algorithm name). Based on the algorithm capability description information, it can be determined that the algorithm is used for fishing detection and needs to have fishing detection capability.

[0057] Exemplarily, the optimized prompt words may be closer to the above-mentioned target business scenario and describe the target business scenario more accurately.

[0058] Exemplarily, when the optimized prompt words are obtained in the above manner, the visual domain tasks of the target business scenario can be performed based on the optimized prompt words using a multimodal large model.

[0059] For example, when an application instruction for an optimized prompt word is detected, the visual domain task of the target business scenario can be performed based on the optimized prompt word using a multimodal large model.

[0060] It can be seen that in Figure 1In the method flow shown, by obtaining the initial prompt word of the target business scenario, and based on the obtained initial prompt word, using the multimodal large model, executing the visual field task of the target business scenario, and obtaining the output event, based on the initial prompt word, a positive learning and optimization initial direction is provided for the multimodal large model, and there is no need to pre-train the multimodal large model for the target business scenario. In the case of cold start of the multimodal large model algorithm, the multimodal large model realizes self-learning of the prompt word. Then, when it is determined that the prompt word optimization conditions are met, the prompt word is optimized based on the initial prompt word, the event picture with annotation information corresponding to the output event, and the algorithm capability description information corresponding to the target business scenario, and the optimized prompt word is obtained. The initial prompt word is optimized using the self-learning ability of the multimodal large model. Since the multimodal large model can learn information in multiple modes such as text and pictures, the prompt word optimization by the multimodal large model can obtain prompt words that are more suitable for the target business scenario, providing technical support for improving the performance of using the multimodal large model to execute the visual field task of the target business scenario.

[0061] In some embodiments, the above-mentioned optimization of prompt words using a multimodal large model based on the initial prompt words, the event picture with annotated information corresponding to the output event, and the algorithm capability description information corresponding to the target business scenario to obtain the optimized prompt words may include:

[0062] The initial prompt words, the event pictures with annotated information corresponding to the output events, and the algorithm capability description information corresponding to the target business scenario are input into the multimodal large model, so that the multimodal large model can learn the text vectors between the initial prompt words and the positive event pictures, as well as the text vectors between the initial prompt words and the false alarm event pictures based on the algorithm capability description information corresponding to the target business scenario, and give a text description that better matches the positive event pictures, and generate optimized prompt words based on the text description.

[0063] Exemplarily, when it is determined that the conditions for prompt word optimization are met, the initial prompt word, the event image with annotation information corresponding to the output event, and the algorithm capability description information corresponding to the target business scenario can be input into the multimodal big model, and the prompt word optimization can be performed by using the multimodal big model's ability to optimize the semantic relationship between different modalities (such as images and texts), such as the ability to handle long-distance dependencies.

[0064] When the multimodal large model receives the above input information, it can determine the algorithm requirements based on the algorithm capability description information corresponding to the target business scenario, and based on the algorithm requirements, convert the event picture into an image vector, convert the initial prompt word into a text vector, and map the image vector and the text vector to a shared embedding space. In this space, the text description and the image content can be matched in the form of vectors, and the prompt word optimization can be achieved by making the prompt word closer to the positive event picture in the space, and making the prompt word further away from the false alarm event picture in the space, so as to obtain a text description that better matches the positive event picture, and then generate the optimized prompt word based on the text description.

[0065] In one example, the optimized prompt word may include an optimized initial prompt word.

[0066] For example, in the fishing detection scenario, the initial prompt words may include: a person by the river, with a fishing rod. The optimized initial prompt words may include: a person sitting by the river, with a fishing rod.

[0067] Taking helmet detection as an example, the initial prompt word may include: helmet. The optimized initial prompt word may include: wearing helmet, not holding helmet, etc.

[0068] In another example, the optimized prompt words may include initial prompt words and newly added prompt words.

[0069] Exemplarily, the newly added prompt word may include a prompt word used to supplement the description of the initial prompt word.

[0070] Still taking the fishing scene as an example, assuming that the initial prompt words include: a person by the river, with a fishing rod, then the newly added prompt words may include a fish bucket and / or a fishing net, etc.

[0071] In yet another example, the optimized prompt words include optimized initial prompt words and newly added prompt words.

[0072] Exemplarily, during the process of optimizing prompt words, on the one hand, the initial prompt words may be optimized to obtain optimized initial prompt words; on the other hand, new prompt words may be obtained, such as prompt words used to supplement the description of the initial prompt words.

[0073] In some embodiments, the large model algorithm cold start prompt word optimization method provided in the embodiments of the present application may also include:

[0074] When it is determined that the prompt word optimization conditions are met, the initial prompt word, the event image with annotation information corresponding to the output event, and the algorithm capability description information corresponding to the target business scenario are input into the multimodal large model, so that the multimodal large model converts the initial prompt word, the event image with annotation information corresponding to the output event, and the algorithm capability description information corresponding to the target business scenario into corresponding vectors, generates a feature vector by comparing and learning the image-text pairs, and uses the comparison function to optimize the weight parameters of the multimodal large model.

[0075] Exemplarily, in order to further improve the performance of the multimodal large model in executing visual domain tasks of the target business scenarios, on the one hand, the prompt words can be optimized in a certain manner to obtain optimized prompt words; on the other hand, the model weight parameters of the multimodal large model can also be optimized to make the multimodal large model more suitable for processing visual domain tasks of the target business scenarios.

[0076] Accordingly, when it is determined that the prompt word optimization conditions are met, the weight parameters of the multimodal large model can be optimized based on the event image with annotation information corresponding to the initial prompt word output event, as well as the algorithm capability description information corresponding to the target business scenario.

[0077] Exemplarily, an event picture with annotation information corresponding to the initial prompt word output event and algorithm capability description information corresponding to the target business scenario can be input into the multimodal large model.

[0078] When the multimodal large model receives the above information, the multimodal large model can convert the input image into an image vector, and convert the input text into a text vector, and map the obtained image vector and text vector into a shared embedding space. The multimodal large model optimizes the similarity of text-image pairs through contrastive learning, so that the distance between the prompt word and the positive event picture in the space is as close as possible, and the distance between the prompt word and the false alarm event picture in the space is as far as possible, so as to optimize the weight parameters of the multimodal large model.

[0079] It should be noted that, in the above-mentioned multimodal large model weight parameter optimization process, the weight parameters actually used by the multimodal large model are still the weight parameters before optimization, and when an application instruction for the optimized multimodal large model weight parameters is detected, the weight parameters need to be updated and the multimodal large model weight parameters before optimization are replaced with the optimized multimodal large model weight parameters.

[0080] Exemplarily, the multimodal large model can use the event images of the above-mentioned output events as materials, reclassify the output events in vector space, and combine the image and text understanding capabilities of the multimodal large model to learn optimized prompt words. The optimized prompt words can, on the one hand, optimize the initial scene description to make it more accurate (that is, the initial prompt words can be optimized), and on the other hand, based on the generalization ability of the multimodal large model, new descriptions of the target business scenarios can also be learned (that is, new prompt words can be added).

[0081] During the prompt word optimization process, the multimodal large model re-learns the vector space to detect new scenario knowledge of the business scenario (i.e., the target business scenario mentioned above) and obtain model weight parameters that are more suitable for the target business scenario.

[0082] In some embodiments, Figure 2 As shown, after obtaining the optimized prompt words, the following may also be included:

[0083] When the number of iterative optimizations of the prompt words does not reach the preset maximum number, the prompt words are optimized again using the multimodal large model according to the optimized prompt words, the event pictures with annotated information corresponding to the output events, and the algorithm capability description information corresponding to the target business scenario, to obtain the re-optimized prompt words;

[0084] When the number of iterative optimizations of the prompt word reaches a preset maximum number, the currently obtained optimized prompt word is determined as the final optimized prompt word.

[0085] For example, in order to improve the optimization effect of prompt words, prompt words may be optimized by iterative optimization.

[0086] Accordingly, in accordance with the method described in the above embodiment, based on the initial prompt word, the event picture with annotation information corresponding to the output event, and the algorithm capability description information corresponding to the target business scenario, the prompt word optimization is performed using a multimodal large model. When the optimized prompt word is obtained, it can be determined whether the current prompt word iterative optimization number is greater than the preset maximum number (which can be set according to actual needs).

[0087] If the number of iterative optimizations of the prompt word does not reach the preset maximum number, the optimized prompt word can be optimized again. In this case, the prompt word can be optimized again based on the optimized prompt word, the event picture with annotation information corresponding to the output event, and the algorithm capability description information corresponding to the target business scenario to obtain the optimized prompt word.

[0088] When the number of iterative optimizations of the prompt word reaches a preset maximum number, the currently obtained optimized prompt word may be determined as the final optimized prompt word.

[0089] Similarly, in an embodiment of the present application, in the process of iteratively optimizing the prompt words, the weight parameters of the multimodal large model can also be iteratively optimized, that is, each time the prompt words are optimized again based on the optimized prompt words, the event pictures with annotation information corresponding to the output events, and the algorithm capability description information corresponding to the target business scenario, the multimodal large model is used to optimize the weight parameters of the multimodal large model, and the optimized multimodal large model weight parameters are obtained.

[0090] In some embodiments, when the optimized prompt words are obtained, the following may also be included:

[0091] Obtain the confidence of the optimized prompt words recommended by the multimodal large model;

[0092] Based on the optimized prompt words and the confidence of the optimized prompt words, a multimodal large model is used to perform visual domain tasks in the target business scenario.

[0093] Exemplarily, in the process of optimizing prompt words using the multimodal large model, the recommendation confidence for each optimized prompt word given by the multimodal large model can be obtained, and the recommendation confidence given by the multimodal large model can be determined as the confidence of the optimized prompt word.

[0094] After obtaining the optimized prompt words in the above manner and determining the confidence of the optimized prompt words, the multimodal large model can be used to perform visual domain tasks in the target business scenario based on the optimized prompt words and the confidence of the optimized prompt words, thereby improving the detection accuracy while ensuring the detection rate.

[0095] Exemplarily, in the process of executing the visual domain task of the target business scenario using the multimodal large model based on the optimized prompt word and the confidence of the optimized prompt word, for any prompt word, the probability that the target detected from the input image is the target corresponding to the prompt word determined by the multimodal large model exceeds the confidence corresponding to the prompt word, then it is determined that the target corresponding to the prompt word is detected. Furthermore, it can be determined whether a specified event exists based on the detected target.

[0096] The method provided by the present application is described above. The device provided by the present application is described below:

[0097] See also Figure 3 , which is a structural diagram of a large model algorithm cold start prompt word optimization device provided in an embodiment of the present application, such as Figure 3 As shown, the large model algorithm cold start prompt word optimization device may include:

[0098] An acquisition unit 310 is configured to acquire an initial prompt word of a target business scenario;

[0099] The execution unit 320 is configured to execute the visual domain task of the target business scenario based on the initial prompt word using a multimodal large model that is not pre-trained for the target business scenario to obtain an output event;

[0100] The optimization unit 330 is configured to optimize the prompt words using a multimodal large model based on the initial prompt words, the event picture with annotation information corresponding to the output event, and the algorithm capability description information corresponding to the target business scenario, when it is determined that the prompt word optimization conditions are met, to obtain optimized prompt words; wherein the annotation information includes true positive or false positive.

[0101] In some embodiments, the initial confidence of the initial prompt word is less than a preset confidence threshold; wherein the confidence of the prompt word is positively correlated with the accuracy of detecting the target corresponding to the prompt word from the image using the multimodal large model.

[0102] In some embodiments, satisfying the prompt word optimization conditions includes: the number of output events reaches a preset number, and / or the duration of executing the visual domain task of the target business scenario using the multimodal large model reaches a preset duration.

[0103] In some embodiments, the optimization unit 330 is specifically configured to input the initial prompt word, the event picture with annotation information corresponding to the output event, and the algorithm capability description information corresponding to the target business scenario into the multimodal large model, so that the multimodal large model learns the text vector between the initial prompt word and the positive event picture, as well as the text vector between the initial prompt word and the false alarm event picture based on the algorithm capability description information corresponding to the target business scenario, and gives a text description that better matches the positive event picture, and generates an optimized prompt word based on the text description.

[0104] In some embodiments, the optimized prompt word includes an optimized initial prompt word;

[0105] or,

[0106] The optimized prompt words include the initial prompt words and the newly added prompt words;

[0107] or,

[0108] The optimized prompt words include the optimized initial prompt words and the newly added prompt words.

[0109] In some embodiments, the optimization unit 330 is also configured to input the initial prompt word, the event picture with annotation information corresponding to the output event, and the algorithm capability description information corresponding to the target business scenario into the multimodal large model when it is determined that the prompt word optimization conditions are met, so that the multimodal large model can convert the initial prompt word, the event picture with annotation information corresponding to the output event, and the algorithm capability description information corresponding to the target business scenario into corresponding vectors, generate feature vectors by comparing and learning image-text pairs, and use the comparison function to optimize the weight parameters of the multimodal large model.

[0110] In some embodiments, the optimization unit 330 is further configured to, after obtaining the optimized prompt word, if the number of iterative optimization times of the prompt word has not reached a preset maximum number, optimize the prompt word again using a multimodal large model based on the optimized prompt word, the event picture with annotation information corresponding to the output event, and the algorithm capability description information corresponding to the target business scenario to obtain the again optimized prompt word; if the number of iterative optimization times of the prompt word reaches a preset maximum number, determine the currently obtained optimized prompt word as the final optimized prompt word.

[0111] In some embodiments, Figure 4 As shown, the large model algorithm cold start prompt word optimization device may also include:

[0112] A determination unit 340 is configured to obtain the confidence of the optimized prompt word recommended by the multimodal large model;

[0113] The execution unit 320 is further configured to execute the visual domain task of the target business scenario using the multimodal large model based on the optimized prompt words and the confidence of the optimized prompt words.

[0114] See also Figure 5 , is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. The electronic device may include a processor 501, a communication interface 502, a memory 503 and a communication bus 504. The processor 501, the communication interface 502 and the memory 503 communicate with each other through the communication bus 504. Among them, the memory 503 stores a computer program; the processor 501 can execute the large model algorithm cold start prompt word optimization method described above by executing the program stored in the memory 503.

[0115] The memory 503 mentioned herein may be any electronic, magnetic, optical or other physical storage device that may contain or store information, such as executable instructions, data, etc. For example, the memory 503 may be: RAM (Radom Access Memory), volatile memory, non-volatile memory, flash memory, storage drive (such as hard disk drive), solid state drive, any type of storage disk (such as optical disk, DVD, etc.), or similar storage media, or a combination thereof.

[0116] The present application also provides a non-transitory machine-readable storage medium storing a computer program, such as Figure 5 The computer program may be stored in the memory 503. Figure 5 The processor 501 in the electronic device is executed to implement the large model algorithm cold start prompt word optimization method described above.

[0117] The present application also provides a computer program stored in a non-transitory machine-readable storage medium, such as Figure 5 The memory 503 in the memory, and when the processor executes the computer program, it prompts the processor 501 to execute the large model algorithm cold start prompt word optimization method described above.

Claims

1. A large model algorithm cold start prompt word optimization method, characterized in that: include: Get the initial prompt words of the target business scenario; According to the initial prompt word, using a multimodal large model that is not pre-trained for the target business scenario, executing a visual domain task of the target business scenario to obtain an output event; When it is determined that the prompt word optimization conditions are met, the prompt word is optimized using the multimodal large model based on the initial prompt word, the event picture with annotation information corresponding to the output event, and the algorithm capability description information corresponding to the target business scenario to obtain the optimized prompt word; wherein the annotation information includes true positive or false positive.

2. The method according to claim 1, characterized in that The initial confidence of the initial prompt word is less than a preset confidence threshold; wherein the confidence of the prompt word is positively correlated with the accuracy of detecting the target corresponding to the prompt word from the picture using the multimodal large model.

3. The method according to claim 1, characterized in that: The conditions for optimizing the prompt words are met, including: the number of output events reaches a preset number, and / or the duration of executing the visual domain task of the target business scenario using the multimodal large model reaches a preset duration.

4. The method according to claim 1, characterized in that The method of optimizing prompt words by using the multimodal large model based on the initial prompt words, the event picture with annotation information corresponding to the output event, and the algorithm capability description information corresponding to the target business scenario to obtain optimized prompt words includes: The initial prompt word, the event picture with annotation information corresponding to the output event, and the algorithm capability description information corresponding to the target business scenario are input into the multimodal large model, so that the multimodal large model learns the text vector between the initial prompt word and the positive event picture, as well as the text vector between the initial prompt word and the false alarm event picture based on the algorithm capability description information corresponding to the target business scenario, and gives a text description that better matches the positive event picture, and generates an optimized prompt word based on the text description.

5. The method according to claim 4, characterized in that The optimized prompt words include the optimized initial prompt words; or, The optimized prompt words include the initial prompt words and the newly added prompt words; or, The optimized prompt words include optimized initial prompt words and newly added prompt words.

6. The method according to claim 1, characterized in that The method further comprises: When it is determined that the prompt word optimization conditions are met, the initial prompt word, the event picture with annotation information corresponding to the output event, and the algorithm capability description information corresponding to the target business scenario are input into the multimodal large model, so that the multimodal large model converts the initial prompt word, the event picture with annotation information corresponding to the output event, and the algorithm capability description information corresponding to the target business scenario into corresponding vectors, generates a feature vector by comparing and learning the image-text pairs, and uses the comparison function to optimize the weight parameters of the multimodal large model to obtain the optimized large model weight parameters.

7. The method according to claim 1, characterized in that After obtaining the optimized prompt word, the method further includes: When the number of iterative optimizations of the prompt words does not reach the preset maximum number, the prompt words are optimized again by using the multimodal large model according to the optimized prompt words, the event picture with annotation information corresponding to the output event, and the algorithm capability description information corresponding to the target business scenario, to obtain the re-optimized prompt words; When the number of iterative optimizations of the prompt word reaches a preset maximum number, the currently obtained optimized prompt word is determined as the final optimized prompt word.

8. The method according to any one of claims 1 to 7, characterized in that: When the optimized prompt words are obtained, it also includes: Obtaining the confidence of the optimized prompt word recommended by the multimodal large model; The visual domain task of the target business scenario is performed using the multimodal large model according to the optimized prompt words and the confidence of the optimized prompt words.

9. A large model algorithm cold start prompt word optimization device, characterized in that: include: An acquisition unit, configured to acquire an initial prompt word of a target business scenario; An execution unit is configured to execute the visual domain task of the target business scenario based on the initial prompt word and use a multimodal large model that is not pre-trained for the target business scenario to obtain an output event; The optimization unit is configured to optimize the prompt words using the multimodal large model based on the initial prompt words, the event picture with annotation information corresponding to the output event, and the algorithm capability description information corresponding to the target business scenario when it is determined that the prompt word optimization conditions are met, so as to obtain the optimized prompt words; wherein the annotation information includes a true positive or a false positive.

10. An electronic device, characterized in that: include: A processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory, used to store computer programs; A processor, for implementing the method according to any one of claims 1 to 8 when executing a program stored in a memory.

Images