Data processing method and device, computer program product and electronic equipment
By inputting real-time images to the content extractor and combining with the large language model, the problem of low output accuracy of the large language model in emergency warning is solved, and the processing and interpretability of multimodal data is improved.
Patent Information
- Application Number
- CN202411978053.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-30
- Publication Date
- 2025-04-29
AI Technical Summary
The large language model has low output accuracy in emergency warning, and the input data can only be single modal data, so the output answer is low.
By obtaining the target content extractor and the target large language model, the real-time image is input into the content extractor, the image features are obtained, and the target input data is determined based on the image characteristics and real-time problems, and input it into the large language model to output the target output data, including theoretical data, interpretation data and answers.
The integration of language text information and visual information is realized, the perception ability of large models on multimodal data is enhanced, the accuracy of reasoning is improved, and the interpretability of large language models in handling actual business is provided.
Smart Images
Figure CN119904688A_ABST
Abstract
Description
Background Art
[0002] In recent years, artificial intelligence has developed rapidly, and the technology has gradually undergone major changes: from the era of small models in various fields (such as computer vision, natural language, search and promotion, etc.) to the current era of large models. The innovation of large model technology has brought new opportunities for change to enterprises. Large models can handle scenarios such as image classification, visual question answering, speech recognition and synthesis, video understanding and generation, and solve problems involving text, pictures, audio, etc. in actual business needs.
[0003] In related technologies, the use of large models in emergency warnings can quickly identify a variety of risk hazards and disaster scenarios, issue instructions to autonomously perform specific tasks, such as automatic alarms, etc., and widely serve emergency management units at all levels and emergency rescue work. However, the input data of large models can only be single-modal data, and the reasoning process of large models is invisible to users. The output answers are low in interpretability, resulting in low accuracy of the output answers.
[0004] Therefore, a new data processing method needs to be provided.
[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention
[0006] The purpose of the present disclosure is to provide a data processing method, a data processing device, a computer program product, and an electronic device, thereby overcoming, at least to a certain extent, the problem of low output accuracy of large language models in related technologies caused by limitations and defects of related technologies.
[0007] According to one aspect of the present disclosure, there is provided a data processing method, comprising:
[0008] Obtain a target content extractor and a target large language model, input a real-time image into the target content extractor, and obtain image features corresponding to the real-time image;
[0009] Determining target input data according to the real-time problem corresponding to the real-time image and the image features;
[0010] The target input data is input into the target large language model, and target output data is output, wherein the output data includes target theoretical data associated with the real-time question, target interpretation data generated based on the target theoretical data, and a target answer associated with the real-time question.
[0011] In an exemplary embodiment of the present disclosure, the step of obtaining a target content extractor and a target large language model includes:
[0012] Acquire a first image included in the target emergency warning sample data, input the first image into a first content extractor, obtain output image features, and determine a first loss function according to the output image features;
[0013] Inputting the output image features and the first input data corresponding to the first image in the target emergency warning sample data into a first large language model to obtain sample output data, and determining a second loss function according to the sample output data;
[0014] According to the first loss function and the second loss function, a target loss function is determined, and the first content extractor and the first large language model are updated according to the target loss function to obtain the target content extractor and the target large language model.
[0015] In an exemplary embodiment of the present disclosure, before acquiring target emergency warning sample data, the method further includes:
[0016] Acquire an emergency warning data set, and organize question data in the emergency warning data set into warning question data, warning image context information, and warning input data of warning question and answer options;
[0017] Organizing the answer data in the emergency warning data set into warning theory data, warning explanation data and warning output data of warning answer data;
[0018] The target emergency warning sample data is generated based on the warning input data and the warning output data.
[0019] In an exemplary embodiment of the present disclosure, determining the first loss function according to the output image feature includes:
[0020] Acquire first image context information corresponding to the first image in the target emergency warning sample data;
[0021] A cross entropy loss is calculated based on the output image features and the first image context information to obtain the first loss function.
[0022] In an exemplary embodiment of the present disclosure, determining the second loss function according to the sample output data includes:
[0023] Acquire first output data corresponding to the first input data in the target emergency warning sample data;
[0024] A cross entropy loss is calculated for the first output data and the sample output data to obtain the second loss function.
[0025] In an exemplary embodiment of the present disclosure, determining target input data according to the real-time question corresponding to the real-time image and the image features includes:
[0026] Determine an early warning question and answer option corresponding to the real-time question, and generate target input data according to the real-time question, the early warning question and answer option, and the image feature.
[0027] In an exemplary embodiment of the present disclosure, inputting the target input data into the target large language model and outputting the target output data includes:
[0028] Inputting the target input data into the target large language model, and outputting early warning theoretical data corresponding to the real-time problem;
[0029] Based on the warning theory data, generating and outputting warning explanation data associated with the real-time problem;
[0030] Based on the warning theory data and the warning explanation data, a warning target answer is obtained and output.
[0031] According to one aspect of the present disclosure, there is provided a data processing device, comprising:
[0032] An image feature extraction module, used to obtain a target content extractor and a target large language model, input a real-time image into the target content extractor, and obtain image features corresponding to the real-time image;
[0033] An input data determination module, used to determine target input data according to the real-time problem corresponding to the real-time image and the image features;
[0034] A data output module is used to input the target input data into the target large language model and output target output data, wherein the output data includes target theoretical data associated with the real-time image, target interpretation data generated based on the target theoretical data, and a target answer associated with the real-time question.
[0035] According to one aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the data processing method described in any of the above exemplary embodiments is implemented.
[0036] According to one aspect of the present disclosure, there is provided an electronic device, including:
[0037] A processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the data processing method described in any of the above exemplary embodiments by executing the executable instructions.
[0038] A data processing method provided by an embodiment of the present disclosure obtains a target content extractor and a target large language model, inputs a real-time image into the target content extractor, and obtains image features corresponding to the real-time image; determines target input data according to a real-time question corresponding to the real-time image and the image features; inputs the target input data into the target large language model, and outputs target output data, wherein the output data includes target theoretical data associated with the real-time question, target interpretation data generated based on the target theoretical data, and a target answer associated with the real-time question. On the one hand, after acquiring the real-time image, the real-time image is input into the target content extractor to obtain the image features corresponding to the real-time image, and the target input data is determined based on the image features and the real-time question, which solves the problem in the related technology that the input data can only be single-modal data, realizes the fusion of language text information and visual information, and enhances the perception ability of the large model to multi-modal data; on the other hand, after obtaining the target input data, the target input data is input into the target large language model, and based on the target theoretical data associated with the real-time question output by the target large language model, the target interpretation data generated based on the theoretical data, and the target answer determined according to the target theoretical data and the target interpretation data, the intermediate process of logical reasoning of the target large language model is explicitly generated, thereby improving the reasoning accuracy of the target large language model; further, the interpretability of the target large language model in processing actual business is provided, and the answers are traceable.
[0039] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.
[0041] Figure 1 The flowchart schematically shows a data processing method according to an exemplary embodiment of the present disclosure.
[0042] Figure 2 A flowchart of a method for obtaining a target content extractor and a target large language model according to an exemplary embodiment of the present disclosure is schematically shown.
[0043] Figure 3A flowchart of a data processing method before acquiring target emergency warning sample data according to an example embodiment of the present disclosure is schematically shown.
[0044] Figure 4 A flowchart of a method for determining a first loss function according to the output image features according to an exemplary embodiment of the present disclosure is schematically shown.
[0045] Figure 5 A flow chart of a method for determining a second loss function according to the sample output data according to an exemplary embodiment of the present disclosure is schematically shown.
[0046] Figure 6 A flowchart of a method for inputting the target input data into the target large language model and outputting target output data according to an exemplary embodiment of the present disclosure is schematically shown.
[0047] Figure 7 The following is a block diagram schematically showing a data processing device according to an exemplary embodiment of the present disclosure.
[0048] Figure 8 An electronic device for implementing the above data processing method according to an exemplary embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION
[0049] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.
[0050] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.
[0051] This exemplary embodiment first provides a data processing method, which can be run on a terminal device; of course, those skilled in the art can also run the method of the present invention on other platforms as required, and this exemplary embodiment does not specifically limit this. Figure 1 As shown, the connection establishment method may include steps S110 to S130:
[0052] Step S110. Obtain a target content extractor and a target large language model, input a real-time image into the target content extractor, and obtain image features corresponding to the real-time image;
[0053] Step S120. Determine target input data according to the real-time problem corresponding to the real-time image and the image features;
[0054] Step S130. Input the target input data into the target large language model, and output target output data, wherein the output data includes target theoretical data associated with the real-time question, target interpretation data generated based on the target theoretical data, and a target answer associated with the real-time question.
[0055] The above-mentioned data processing method obtains a target content extractor and a target large language model, inputs a real-time image into the target content extractor, and obtains image features corresponding to the real-time image; determines target input data according to the real-time question corresponding to the real-time image and the image features; inputs the target input data into the target large language model, and outputs target output data, wherein the output data includes target theoretical data associated with the real-time question, target interpretation data generated based on the target theoretical data, and target answer associated with the real-time question. On the one hand, after acquiring the real-time image, the real-time image is input into the target content extractor to obtain the image features corresponding to the real-time image, and the target input data is determined based on the image features and the real-time question, which solves the problem in the related technology that the input data can only be single-modal data, realizes the fusion of language text information and visual information, and enhances the perception ability of the large model to multi-modal data; on the other hand, after obtaining the target input data, the target input data is input into the target large language model, and based on the target theoretical data associated with the real-time question output by the target large language model, the target interpretation data generated based on the theoretical data, and the target answer determined according to the target theoretical data and the target interpretation data, the intermediate process of logical reasoning of the target large language model is explicitly generated, thereby improving the reasoning accuracy of the target large language model; further, the interpretability of the target large language model in processing actual business is provided, and the answers are traceable.
[0056] Hereinafter, each step involved in the data connection method of the exemplary embodiment of the present disclosure is explained and illustrated in detail.
[0057] First, the application scenario and purpose of the exemplary embodiment of the present disclosure are explained and illustrated. Specifically, the exemplary embodiment of the present disclosure can be applied to a large language model to perform fusion reasoning on language text and visual information and explicitly generate an intermediate process of logical reasoning of the large language model.
[0058] In the present disclosure, based on the trained target content extractor and the target large language model, a real-time image is obtained, the real-time image is input into the target content extractor, and the image features of the real-time image are obtained based on the target content extractor. After the image features of the real-time image are obtained, the target input data is determined based on the factual image features, the real-time question input by the user, and the question and answer options, the target input data is input into the target large language model, the target theoretical data associated with the real-time question is obtained through the target large language model, the target explanation data is generated based on the target theoretical data, and the target answer is generated according to the target theoretical data and the target explanation data.
[0059] Next, steps S110 to S130 are explained and illustrated in detail.
[0060] In step S110, a target content extractor and a target large language model are obtained, and a real-time image is input into the target content extractor to obtain image features corresponding to the real-time image.
[0061] In this example embodiment, the target content extractor may be Vit (Vision Transformer) or an image classification task model. In this example embodiment, no limitation is imposed on the target content extractor. The target large language model may be a deep learning model based on the Transformer architecture, specifically Qwen, or other models. In this example embodiment, no limitation is imposed on the target large language model. The real-time image may be a live photo of the target scene, which may be a real-time photo taken when a fire occurs in the target scene. In this example embodiment, no limitation is imposed on the real-time image. After obtaining the target content extractor and the target large language model, the real-time image may be input into the target content extractor to extract image features of the real-time image.
[0062] In this example embodiment, reference Figure 2 As shown, the step of obtaining a target content extractor and a target large language model includes:
[0063] Step S210. Obtain a first image included in the target emergency warning sample data, input the first image into a first content extractor, obtain output image features, and determine a first loss function according to the output image features;
[0064] Step S220: Input the output image features and the first input data corresponding to the first image in the target emergency warning sample data into the first large language model to obtain sample output data, and determine the second loss function according to the sample output data;
[0065] Step S230. Determine a target loss function according to the first loss function and the second loss function, and update the first content extractor and the first large language model according to the target loss function to obtain the target content extractor and the target large language model.
[0066] In the following, steps S210 to S230 will be further explained and illustrated. Specifically, a first image included in the target emergency warning sample data is obtained, the first image is input into a first content extractor, and the image features of the first image are output based on the first content extractor. After the image features of the first image are obtained, the first loss function Loss of the first content extractor can be determined according to the image features of the first image. V At the same time, the output image feature of the first image and the first input data corresponding to the first image in the target emergency warning sample data are input into the first large language model to obtain sample output data. After obtaining the sample output data, the second loss function Loss of the first large language model can be determined according to the sample output data. L After obtaining the first loss function and the second loss function, the first loss function and the second loss function can be weighted to obtain the target loss function Loss = αLoss V +(1-α)Loss L , where α = 0.4. After obtaining the target loss function, the first content extractor and the first large language model can be back-propagated and optimized using the AdamW optimizer according to the target loss function until the model converges to obtain the target content extractor and the target large language model.
[0067] In this example embodiment, reference Figure 3 As shown, before obtaining the target emergency warning sample data, the method further includes:
[0068] Step S310. Acquire an emergency warning data set, and organize the question data in the emergency warning data set into warning question data, warning image context information, and warning input data of warning question and answer options;
[0069] Step S320. Organize the answer data in the emergency warning data set into warning theory data, warning explanation data and warning output data of warning answer data;
[0070] Step S330: Generate the target emergency warning sample data based on the warning input data and the warning output data.
[0071] In the following, step S310-step S330 will be further explained and illustrated. Specifically, an emergency warning data set is obtained, and the emergency warning data set can be a multimodal emergency warning question-answer pair data set. After the emergency warning data set is obtained, the problem data in the emergency warning data set can be organized into three-segment warning input data, and the three-segment warning input data includes warning problem data, warning image context information, and warning question-answer answer options. Among them, the warning problem data can be a question input by a user, the warning image context information is an image feature corresponding to the image, and the warning question-answer answer options are answer options corresponding to the warning problem data. At the same time, the answer data in the emergency warning data set is organized into three-segment warning output data, and the three-segment warning output data includes warning theory data, warning explanation data, and warning answer data. Among them, the warning theory data can be an objective knowledge principle related to the warning, the warning explanation data can be an explanation of the objective knowledge principle combined with the problem itself, and the warning answer data can be the answer corresponding to the warning problem data. After obtaining the warning input information and the warning output information, the target emergency warning sample data can be generated according to the three-segment warning input information and the warning output information.
[0072] In this example embodiment, reference Figure 4 As shown, determining the first loss function according to the output image feature includes:
[0073] Step S410. Acquire first image context information corresponding to the first image in the target emergency warning sample data;
[0074] Step S420: Perform cross entropy loss calculation based on the output image features and the first image context information to obtain the first loss function.
[0075] In the following, step S410 and step S420 are further explained and illustrated. Specifically, the first image context information corresponding to the first image in the target emergency warning sample data is obtained, and the cross entropy loss calculation is performed on the output image features and the first image context information to obtain the first loss function Loss v =H v (p v , Q v )=-∑i P v (i)logQ v (i), where p v is the true category distribution, Q v is the probability distribution predicted by the model, H v is the cross entropy loss.
[0076] In this example embodiment, reference Figure 5 As shown, determining the second loss function according to the sample output data includes:
[0077] Step S510: Obtain the first output data corresponding to the first input data in the target emergency warning sample data:
[0078] Step S520: Perform cross entropy loss calculation on the first output data and the sample output data to obtain the second loss function.
[0079] In the following, step S510 and step S520 will be further explained and illustrated. Specifically, the first output data corresponding to the first input data in the target emergency warning sample data is obtained, and the cross entropy loss calculation is performed on the sample output data output by the first large language model and the first output data in the target emergency warning sample data to obtain the second loss function Loss L =H L (p L , Q L )=-∑ i P L (i)logQ L (i), where p L is the real category classification, Q L is the probability distribution predicted by the model, H L is the cross entropy loss.
[0080] In step S120, target input data is determined according to the real-time question corresponding to the real-time image and the image features.
[0081] After obtaining the image features corresponding to the real-time image, the target input data can be determined according to the image features. The step of determining the target input data according to the real-time question corresponding to the real-time image and the image features includes:
[0082] Determine an early warning question and answer option corresponding to the real-time question, and generate target input data according to the real-time question, the early warning question and answer option, and the image feature.
[0083] Specifically, the early warning question and answer options corresponding to the real-time question are determined, and the target input data is generated based on the real-time question, the image features of the real-time image, and the early warning question and answer options.
[0084] In step S130, the target input data is input into the target large language model, and target output data is output, wherein the output data includes target theoretical data associated with the real-time question, target interpretation data generated based on the target theoretical data, and a target answer associated with the real-time question.
[0085] In this example embodiment, after the target input data is generated, the target input data can be input into the target large language model to obtain the target output data of the target large language model. The target output data includes target theoretical data associated with the real-time question, target interpretation data generated based on the target theoretical data, and target answers associated with the real-time question.
[0086] refer to Figure 6 As shown, the inputting the target input data into the target large language model and outputting the target output data comprises:
[0087] Step S610. Input the target input data into the target large language model, and output the early warning theory data corresponding to the real-time problem;
[0088] Step S620. Based on the warning theory data, generate and output warning explanation data associated with the real-time problem;
[0089] Step S630. Based on the warning theory data and the warning explanation data, obtain and output the warning target answer.
[0090] Below, step S610-step S630 will be further explained and illustrated. Specifically, the generated target input data is input into the target large language model, and the warning theory data corresponding to the real-time problem is generated and output based on the target large language model. After generating the warning theory data, the target large language model generates warning explanation data associated with the real-time problem in combination with the warning theory data, and outputs the warning explanation data. Finally, the target large language model determines the warning target answer corresponding to the real-time problem based on the warning theory data and the warning explanation data, and outputs the warning target answer.
[0091] The data processing method provided by the example embodiment of the present disclosure has at least the following advantages: on the one hand, the terminal device sends an attachment request message to the mobility management entity, the attachment request message includes a public data network connection establishment request, and accepts the indication information sent by the mobility management entity. After the terminal device receives the indication information, it sends a request to the mobility management entity according to the indication information, which solves the problem in the related art that the terminal device can only send the attachment request after the satellite is connected to the ground and covers the terminal device; on the other hand, the terminal device can directly send a request to the mobility management entity after receiving the indication information, which improves the efficiency of sending the attachment request.
[0092] The exemplary embodiment of the present disclosure also provides a data processing device, referring to Figure 7 As shown, it may include: an image feature extraction module 710, an input data determination module 720 and a data output module 730. Among them:
[0093] An image feature extraction module 710 is used to obtain a target content extractor and a target large language model, input a real-time image into the target content extractor, and obtain image features corresponding to the real-time image;
[0094] An input data determination module 720, configured to determine target input data according to a real-time question corresponding to the real-time image and the image features;
[0095] The data output module 730 is used to input the target input data into the target large language model and output target output data, wherein the output data includes target theoretical data associated with the real-time image, target interpretation data generated based on the target theoretical data, and target answer associated with the real-time question.
[0096] The specific details of each module in the above data processing device have been described in detail in the corresponding data processing method, so they will not be repeated here.
[0097] In an exemplary embodiment of the present disclosure, the image feature extraction module includes:
[0098] A first loss function determination module is used to obtain a first image included in the target emergency warning sample data, input the first image into a first content extractor, obtain output image features, and determine a first loss function according to the output image features;
[0099] A second loss function determination module is used to input the output image features and the first input data corresponding to the first image in the target emergency warning sample data into a first large language model to obtain sample output data, and determine a second loss function according to the sample output data;
[0100] A target model generation module is used to determine a target loss function according to the first loss function and the second loss function, and to update the first content extractor and the first large language model according to the target loss function to obtain the target content extractor and the target large language model.
[0101] In an exemplary embodiment of the present disclosure, the first loss function determination module includes:
[0102] A question data organization module is used to obtain an emergency warning data set, and organize the question data in the emergency warning data set into warning question data, warning image context information, and warning input data of warning question and answer options;
[0103] An answer data organization module, used for organizing the answer data in the emergency warning data set into warning theory data, warning explanation data and warning output data of warning answer data;
[0104] The target sample generation module is used to generate the target emergency warning sample data based on the warning input data and the warning output data.
[0105] In an exemplary embodiment of the present disclosure, the first loss function determination module includes:
[0106] A context information acquisition module, used to acquire first image context information corresponding to the first image in the target emergency warning sample data;
[0107] The first loss function generating module is used to perform cross entropy loss calculation based on the output image features and the first image context information to obtain the first loss function.
[0108] In an exemplary embodiment of the present disclosure, the second loss function determination module includes:
[0109] An output data acquisition module, used to acquire first output data corresponding to the first input data in the target emergency warning sample data;
[0110] The second loss function generating module is used to perform cross entropy loss calculation on the first output data and the sample output data to obtain the second loss function.
[0111] In an exemplary embodiment of the present disclosure, the input data determination module includes:
[0112] The target input data determination module is used to determine the early warning question and answer options corresponding to the real-time question, and generate target input data according to the real-time question, the early warning question and answer options and the image features.
[0113] In an exemplary embodiment of the present disclosure, the data output module includes:
[0114] A theoretical data output module, used for inputting the target input data into the target large language model, and outputting early warning theoretical data corresponding to the real-time problem;
[0115] An explanation data output module, used to generate and output the warning explanation data associated with the real-time problem based on the warning theoretical data;
[0116] The target answer output module is used to obtain and output the warning target answer based on the warning theoretical data and the warning explanation data.
[0117] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.
[0118] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.
[0119] In an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.
[0120] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".
[0121] Refer to the following Figure 8 The electronic device 800 according to this embodiment of the present disclosure is described. Figure 8 The electronic device 800 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0122] like Figure 8As shown, the electronic device 800 is in the form of a general computing device. The components of the electronic device 800 may include, but are not limited to: the at least one processing unit 810, the at least one storage unit 820, a bus 830 connecting different system components (including the storage unit 820 and the processing unit 810), and a display unit 840.
[0123] The storage unit stores program codes, which can be executed by the processing unit 810, so that the processing unit 810 performs the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification. For example, the processing unit 810 can perform the following steps: Figure 1 Step S110 shown in: obtaining a target content extractor and a target large language model, inputting a real-time image into the target content extractor, and obtaining image features corresponding to the real-time image; S120: determining target input data according to the real-time question corresponding to the real-time image and the image features; S130: inputting the target input data into the target large language model, and outputting target output data, wherein the output data includes target theoretical data associated with the real-time question, target interpretation data generated based on the target theoretical data, and target answer associated with the real-time question.
[0124] The storage unit 820 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 8201 and / or a cache memory unit 8202 , and may further include a read-only memory unit (ROM) 8203 .
[0125] The storage unit 820 may also include a program / utility 8204 having a set (at least one) of program modules 8205, such program modules 8205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.
[0126] Bus 830 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.
[0127] The electronic device 800 may also communicate with one or more external devices 900 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 800, and / or may communicate with any device that enables the electronic device 800 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 850. In addition, the electronic device 700 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 860. As shown, the network adapter 860 communicates with other modules of the electronic device 800 via a bus 830. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 800, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.
[0128] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.
[0129] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present specification is stored. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary implementations of the present disclosure described in the above "Exemplary Method" section of the present specification.
[0130] According to the program product for implementing the above method in the embodiment of the present disclosure, it can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.
[0131] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
[0132] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.
[0133] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.
[0134] Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).
[0135] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.
[0136] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.
Claims
1. A data processing method, characterized in that: include: Obtain a target content extractor and a target large language model, input a real-time image into the target content extractor, and obtain image features corresponding to the real-time image; Determining target input data according to the real-time problem corresponding to the real-time image and the image features; The target input data is input into the target large language model, and target output data is output, wherein the output data includes target theoretical data associated with the real-time question, target interpretation data generated based on the target theoretical data, and a target answer associated with the real-time question.
2. The method according to claim 1, characterized in that The step of obtaining a target content extractor and a target large language model includes: Acquire a first image included in the target emergency warning sample data, input the first image into a first content extractor, obtain output image features, and determine a first loss function according to the output image features; Inputting the output image feature and the first input data corresponding to the first image in the target emergency warning sample data into a first large language model to obtain sample output data, and determining a second loss function according to the sample output data; According to the first loss function and the second loss function, a target loss function is determined, and the first content extractor and the first large language model are updated according to the target loss function to obtain the target content extractor and the target large language model.
3. The method according to claim 2, characterized in that Before obtaining the target emergency warning sample data, the method further includes: Acquire an emergency warning data set, and organize question data in the emergency warning data set into warning question data, warning image context information, and warning input data of warning question and answer options; Organizing the answer data in the emergency warning data set into warning theory data, warning explanation data and warning output data of warning answer data; The target emergency warning sample data is generated based on the warning input data and the warning output data.
4. The method according to claim 2, characterized in that: The determining of the first loss function according to the output image feature comprises: Acquire first image context information corresponding to the first image in the target emergency warning sample data; A cross entropy loss is calculated based on the output image features and the first image context information to obtain the first loss function.
5. The method according to claim 2, characterized in that: The determining a second loss function according to the sample output data comprises: Acquire first output data corresponding to the first input data in the target emergency warning sample data; A cross entropy loss is calculated for the first output data and the sample output data to obtain the second loss function.
6. The method according to claim 1, characterized in that The step of determining target input data according to the real-time problem corresponding to the real-time image and the image features includes: Determine an early warning question and answer option corresponding to the real-time question, and generate target input data according to the real-time question, the early warning question and answer option, and the image feature.
7. The method according to claim 1, characterized in that The step of inputting the target input data into the target large language model and outputting target output data comprises: Inputting the target input data into the target large language model, and outputting early warning theoretical data corresponding to the real-time problem; Based on the warning theory data, generating and outputting warning explanation data associated with the real-time problem; Based on the warning theory data and the warning explanation data, a warning target answer is obtained and output.
8. A data processing device, characterized in that: include: An image feature extraction module, used to obtain a target content extractor and a target large language model, input a real-time image into the target content extractor, and obtain image features corresponding to the real-time image; An input data determination module, used to determine target input data according to the real-time problem corresponding to the real-time image and the image features; A data output module is used to input the target input data into the target large language model and output target output data, wherein the output data includes target theoretical data associated with the real-time image, target interpretation data generated based on the target theoretical data, and a target answer associated with the real-time question.
9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.
10. An electronic device, characterized in that: include: Processor; and A memory for storing executable instructions of the processor; wherein the processor is configured to perform the method of any one of claims 1-7 by executing the executable instructions.