Large language model-based event processing method and apparatus, device and medium

Abstract

The present application provides a large language model-based event processing method and apparatus, a device and a medium. The large language model-based event processing method comprises: acquiring a question input by a user; acquiring pre-generated event information; fusing the question and the event information to obtain a fusion result; and inputting the fusion result into a pre-trained large language model to obtain reply content corresponding to the question.

Description

Event processing method, device, equipment and medium based on large language model

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on October 27, 2023, with application number 202311416014.7, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of large models, such as artificial intelligence, large language models and human-computer interaction, and in particular to event processing methods, devices, equipment and media based on large language models. Background Art

[0003] Intelligent systems enhance the refinement and intelligence of urban governance through comprehensive perception, interconnection, analysis, response, and application of urban operational trends. They connect key urban elements, enable efficient allocation of public resources, and precisely address urban events.

[0004] However, the accuracy and speed of intelligent systems in responding to questions still need to be improved.

[0005] Summary of the Invention

[0006] The present application provides an event processing method, apparatus, device, and medium based on a large language model.

[0007] According to one aspect of the present application, an event processing method based on a large language model is provided, comprising: obtaining a question input by a user; obtaining pre-generated event information; fusing the input question with the event information to obtain a fusion result; and inputting the fusion result into a pre-trained large language model to obtain a reply content corresponding to the input question.

[0008] According to one aspect of the present application, an event processing device based on a large language model is provided, comprising: an input question acquisition module, configured to obtain questions input by a user; an event information acquisition module, configured to obtain pre-generated event information; an event question fusion module, configured to fuse the input question and the event information to obtain a fusion result; and a reply processing module, configured to input the fusion result into a pre-trained large language model to obtain reply content corresponding to the input question.

[0009] According to another aspect of the present application, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the above-mentioned event processing method based on a large language model.

[0010] According to another aspect of the present application, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the above-mentioned event processing method based on the large language model.

[0011] According to another aspect of the present application, a computer program product is provided, including a computer program, which implements the above-mentioned event processing method based on a large language model when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] FIG1 is a flowchart of an event processing method based on a large language model provided in an embodiment of the present application;

[0013] FIG2 is a flowchart of another event processing method based on a large language model provided in an embodiment of the present application;

[0014] FIG3 is a flowchart of another event processing method based on a large language model provided in an embodiment of the present application;

[0015] FIG4 is a scene diagram of a method for generating event information provided by an embodiment of the present application;

[0016] FIG5 is a scenario diagram of a training method based on a large language model provided in an embodiment of the present application;

[0017] FIG6 is a schematic diagram of a reply content provided in an embodiment of the present application;

[0018] FIG7 is a schematic diagram of the structure of an event processing device based on a large language model provided in an embodiment of the present application;

[0019] FIG8 is a structural diagram of an event information acquisition module provided in an embodiment of the present application;

[0020] FIG9 is a structural diagram of a target frame detection unit provided in an embodiment of the present application;

[0021] FIG10 is a structural diagram of an event problem fusion module provided in an embodiment of the present application;

[0022] FIG11 is a structural diagram of another event problem fusion module provided in an embodiment of the present application;

[0023] FIG12 is a structural diagram of another event processing device based on a large language model provided in an embodiment of the present application;

[0024] FIG13 is a structural diagram of another event processing device based on a large language model provided in an embodiment of the present application;

[0025] FIG14 is a block diagram of an electronic device for executing an event processing method based on a large language model provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] The following describes exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding. They should be considered as merely exemplary. For the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0027] Figure 1 is a flowchart of an event processing method based on a large language model provided in an embodiment of the present application. This embodiment is applicable to situations where a large language model responds to questions based on collected events. The method of this embodiment can be executed by an event processing device based on a large language model. The device can be implemented in software and / or hardware and configured in an electronic device with certain data computing capabilities. The electronic device can be a client device or a server device. Client devices include mobile phones, tablet computers, vehicle-mounted terminals, and desktop computers.

[0028] S101: Obtain questions input by the user.

[0029] A question may refer to natural language content input by a user during human-computer interaction. The user may input data in at least one of the following formats: text, image, voice, or video. The directly input data is recognized to obtain the question text. The client may receive the user input data and send it to an electronic device in an embodiment of the present application. The electronic device then obtains the question text based on the user input data. In one embodiment, the question input by the user is the question text.

[0030] S102: Acquire pre-generated event information.

[0031] Event information refers to information collected about real events that have already occurred or occurred. Events can occur within a specific area, for example, within a city. For example, events can be either abnormal or normal. For example, events can be traffic incidents, environmental incidents, business incidents, or urban management incidents. Events can be specific to a specific field or comprehensive events across multiple fields.

[0032] Events can be collected and aggregated. The number of collected event information must be at least one. Events can be perceived data from the real world. Subsequent processing based on perceived event information can improve the real-time nature of event processing. All operations such as event collection, processing, and display are legally and compliantly authorized.

[0033] S103: Fusing the question and the event information to obtain a fusion result.

[0034] The fusion result is the final input to the large language model. The fusion result includes question and event information. The event information is used to enrich the context of the acquired question and can serve as supplementary content for the question.

[0035] S104: Input the fusion result into a pre-trained large language model to obtain a response content corresponding to the question.

[0036] The large language model is used to understand natural language content and generate corresponding responses. In one embodiment, the large language model is a question-answering model, which is used to answer questions and generate responses to feedback to users. The question and event information are input into the large language model, allowing the large language model to understand the question and the contextual event information, thereby deriving more accurate content from the event information and generating more accurate responses.

[0037] In one example, the event information includes a traffic congestion event in City A on that day, and the question is: What is the traffic handling strategy for City A on that day? The response content includes: The traffic resource allocation strategy for the congested road section in City A within a certain time period on that day, and the traffic resource allocation strategy for the remaining road sections within the remaining time period.

[0038] According to the technical solution of the present application, by obtaining pre-generated event information and fusing it with the question input by the user, a fusion result is obtained, and the fusion result is input into a large language model for question understanding and generation, and the reply content corresponding to the question is obtained. The context associated with the question can be increased, and events can be aggregated. Question replies are given based on the aggregated events, which can improve the prediction accuracy of the reply content, especially for application scenarios of question text, which can improve the processing accuracy of the reply text and increase the speed of question replies.

[0039] FIG2 is a flowchart of another event processing method based on a large language model provided in an embodiment of the present application, which is illustrated based on the above technical solution and can be combined with the above multiple implementation methods.

[0040] S201: Obtain a preset scene event type.

[0041] The scene event type is used to distinguish collected events. Scene event types are typically represented by keywords. Scene events can be typical events, such as those occurring during urban management or city operations. For example, the scene event type could be a traffic accident, another could be parking in a parking lot, or another could be flooding.

[0042] The scenario event type can be pre-entered by the user based on the problem they need to solve. The first user can enter the scenario event type, and the second user can enter the problem. Alternatively, the same user can enter both the scenario event type and the problem. Scenario event types are typically pre-defined based on the problem to be solved. For example, if a user wants to know about city traffic conditions, they can set the scenario event type to traffic congestion, restricted traffic sections, or traffic accidents, etc. Event information related to the scenario event type can be collected and generated from real-time city data. This allows questions to be asked based on the event information related to the scenario event type.

[0043] S202: Filter the scene images corresponding to the scene event type from the multiple collected images.

[0044] Among the multiple captured images, some are unrelated to the scene event type. These unrelated images do not contain content related to the scene event type. For example, in the previous example, the scene event type is a traffic accident. The captured images include images of small animals, two vehicles colliding, and park surveillance images. The two-vehicle collision image is a scene image corresponding to the traffic accident, while the other images are unrelated.

[0045] Descriptive content recognition can be performed on multiple images to determine the descriptive information corresponding to each image. Images with descriptive information that is the same or similar to the scene event type can be identified as scene images. For another example, text describing the scene event type can be input into a pre-trained deep learning model for content matching to obtain scene images corresponding to the scene event type. Other scenarios are also possible and are not limited to this.

[0046] In one embodiment, the event processing method based on the large language model further includes: obtaining a real-time video stream and acquisition information of the real-time video stream; obtaining multiple acquired images from the real-time video stream; determining acquisition information of each acquired image based on the acquisition information of the real-time video stream; wherein the acquisition information includes an acquisition device identifier, an acquisition device location, and an acquisition time.

[0047] A real-time video stream may refer to a video stream captured in real time. The real-time video stream is intercepted or sampled to obtain multiple captured images. The image capture time is determined based on the capture information of the real-time video stream. The capture information of the real-time video stream may include the identifier of the capture device used to capture the real-time video stream, the capture device location, and the capture time. The image capture device identifier is the same as the capture device identifier of the real-time video stream, and the image capture device location is the same as the capture device location of the real-time video stream. The time of each frame recorded in the real-time video stream, i.e., the image capture time, can be determined based on the order or time of the images in the real-time video stream.

[0048] By acquiring the collected image from the real-time video stream and determining the collection information of the collected image based on the collection information of the real-time video stream, the real-time nature of the event can be improved, thereby improving the event perception speed.

[0049] In one embodiment, the collection device includes a traffic monitoring camera within a city; the collection device location includes a shooting location of the traffic monitoring camera.

[0050] A traffic monitoring camera is configured to monitor road anomalies. The acquisition device location may include the location of the traffic monitoring camera and / or the location of the image. For example, the acquisition device location may be intersection A. In one example, the problem may be finding the road section with the highest traffic congestion rate.

[0051] By setting the application scenario to traffic monitoring events, traffic events in urban scenes can be summarized, and questions about traffic monitoring events can be answered on this basis. Real-time urban traffic events can be extracted and summarized, reducing the difficulty of searching and screening urban traffic events and improving the accuracy of answering questions about urban traffic events.

[0052] S203: Detect a target frame corresponding to the scene event type in the scene image.

[0053] The target frame may refer to the image content area corresponding to the scene event type in the scene image. The image content type within the target frame is the same as or similar to the scene event type. For example, if the scene event type is "uncivilized pet keeping," the image within the target frame may be an image of an unleashed pet dog on the road. The scene image is screened to be relevant to the scene event type, and the target frame within the scene image corresponds to the scene event type.

[0054] In one embodiment, detecting a target box corresponding to the scene event type in the scene image includes: performing feature extraction on the text of the scene event type to obtain scene text features; performing feature extraction on the scene image to obtain multiple scene image features; screening target image features similar to the scene text features among the multiple scene image features; detecting an alternative box corresponding to the scene event type based on the target image features; calculating the similarity between the alternative box corresponding to the scene event type and the scene text features; calculating the confidence of the alternative box corresponding to the scene event type based on the similarity between the alternative box corresponding to the scene event type and the scene text features; and determining, based on the confidence of the alternative box corresponding to the scene event type, a target box corresponding to the scene event type in the alternative box corresponding to the scene event type.

[0055] The text of the scene event type may refer to the content of the text being the scene event type. Feature extraction is performed on the text of the scene event type to obtain scene text features. For example, a bidirectional encoder representation from transformers (BERT) may be used as a text encoder to extract features from the text of the scene event type to obtain scene text features. Feature extraction is performed on the scene image to obtain multiple scene image features. For example, a vision transformer model may be used as an image encoder to extract features from the scene image to obtain multiple scene image features.

[0056] The similarity between the scene text feature and the scene image feature is used to indicate the degree of similarity between the content represented by the text feature and the content of the image feature. The scene text feature and the scene image feature can be input into a cross-modal query selection network to obtain the similarity between the scene text feature and each scene image feature. Based on the similarity, the most similar scene image features are screened and determined as multiple target image features. For example, among 900 scene image features, the 300 scene image features with the highest similarity are screened, and each of the 300 scene image features is determined as a target image feature. The target image feature is input into the decoder to obtain an alternative box, for example, the position of the alternative box is obtained, and the coordinates of the four vertices of the rectangle of the alternative box are usually used to represent the position of the alternative box. For each alternative box, the similarity between the scene text feature and the alternative box is calculated and used as the confidence of the alternative box. For example, the similarity between the scene image feature of the alternative box and the scene text feature can be determined as the similarity between the scene text feature and the alternative box.

[0057] By establishing the matching capability between images and texts for pre-set texts of any scene type, open-lexicon target perception is achieved. Based on the input image and the text description to be perceived, the area corresponding to the text description is found in the image, and the coordinates of the four points of the boundary of the area are selected to achieve open-lexicon target detection. This allows the system to accept any text description as input, expand the number of target detection categories to near infinity, enrich the content of scene types, and increase application scenarios.

[0058] S204: Acquire acquisition information of the scene image.

[0059] The acquisition information of a scene image may include at least one of the following: the time, location, method, content, and range of the scene image acquisition. When acquiring multiple images, the acquisition information of each image can be acquired simultaneously. Typically, the acquisition information of each image is recorded during image acquisition.

[0060] S205: Generate event information according to the scene event type, the acquisition information of the scene image, and the target frame.

[0061] The generated event information corresponds to the scene event type. The scene image's captured information is used to determine the content, status, and process of the event represented by the event information. The target frame is used to confirm the basis of the event information, indicating that the event information actually exists. By determining the event information based on the scene event type, captured information, and target frame, we can record the event-related information from multiple dimensions, enriching the event content. The target frame can also be used to trace the source, facilitating the location and process of the event.

[0062] S206: Obtain the question input by the user.

[0063] S207: Obtain pre-generated event information from the generated multiple event information.

[0064] Pre-generate multiple event information. For multiple event information, you can obtain all generated event information or filter some of the generated event information. The filtering condition can be time or similarity. If no event information has been generated, the pre-generated event information can be left blank. The order of S201-S205 and S206 can be reversed. Typically, S206 is executed after S201-S205 are completed.

[0065] S208: Fusing the question and the event information to obtain a fusion result.

[0066] In one embodiment, the question and the event information are fused to obtain a fusion result, including: obtaining standard prompt information, the standard prompt information including a slot for the event information and a question slot; filling the event information into the slot for the event information; and filling the question into the question slot to obtain a fusion result.

[0067] A standard prompt is a textual prompt input to the model. It contains keywords and context for the user's query or question, enabling the model to better understand the user's intent and provide a more accurate response. Standard prompts are used to add event information and questions to form the input data for the large language model. The event information slot is used to fill in the event information, and the question slot is used to fill in the question. The fusion result can be the text obtained after the standard prompt is filled in with the event information and question.

[0068] In one example, the standard prompt message may be: You are now the mayor's assistant. The following are abnormal events that occurred in the city today: {event information}, please handle {problem}.

[0069] By adapting standard prompt information to different questions and event information, the large language model can quickly understand questions and event information based on the prompt template and generate corresponding responses to the questions. This omits the training step of the large language model, which requires it to understand the semantics of different scenarios for different questions in order to generate corresponding responses. This quickly implements the tuning process of the large language model and reduces the labor cost of tuning and updating the large language model. At the same time, it can quickly obtain the corresponding prompt template according to the question in real time, improve the real-time update performance of the large language model, and increase the universality and versatility between the large language model and the question.

[0070] S209: Input the fusion result into a pre-trained large language model to obtain a response content corresponding to the question.

[0071] According to the technical solution of the present application, image events are acquired by capturing multiple images, screening out scene images corresponding to scene event types, and detecting target frames corresponding to scene event types in scene images. Event information is generated based on the acquisition information of the scene images, the scene event types, and the target frames. Events can be generated realistically and in real time, thereby improving the efficiency and authenticity of event acquisition.

[0072] FIG3 is a flowchart of another event processing method based on a large language model provided in an embodiment of the present application, which is explained based on the above technical solution and can be combined with the above multiple implementation methods. The number of pre-generated event information is multiple, and the fusion of the question and the event information is, for example, as follows: filtering the multiple event information obtained to obtain target information corresponding to the question; fusion of the input question with the target information; the number of target information is at least one.

[0073] S301: Obtain a question input by a user.

[0074] S302: Acquire pre-generated event information, where the number of pre-generated event information is multiple.

[0075] S303: Filter the acquired multiple event information to obtain target information corresponding to the problem.

[0076] The target information is event information related to the question. For example, the question is about flooding, and the event information is about uncivilized pet keeping. This event information is unrelated to the question. Multiple event information may be obtained, and some of this information may be unrelated to the question. The event information can be filtered to retain only the content related to the question, thereby reducing redundant input data.

[0077] S304: Fuse the question with the target information to obtain a fusion result; the number of the target information is at least one.

[0078] The number of target information obtained by screening can be multiple. All target information can be integrated with the input question to enrich the input data.

[0079] In one embodiment, standard prompt information is used to add target information and questions to form the input data for the large language model. The event information slot is used to fill in the target information, and the question slot is used to fill in the question. The fusion result can be the text obtained after the standard prompt information is filled in with the target information and question.

[0080] As in the previous example, the standard prompt message may be: You are now the mayor's assistant, and the following are abnormal events that occurred in the city today: {target information}, please handle {problem}.

[0081] S305: Input the fusion result into a pre-trained large language model to obtain a response content corresponding to the question.

[0082] In one embodiment, the pre-trained large language model is determined by instruction tuning of a large language model for processing general tasks, and the large language model for processing general tasks is obtained through unsupervised training, labeled supervised fine-tuning, and human feedback reinforcement learning training.

[0083] Large language models for general-purpose tasks can be pre-trained and fine-tuned to better suit specific application scenarios, regardless of application domain or business scenario. These models can be used to implement general-purpose understanding and generation capabilities.

[0084] For large language models designed to handle general tasks, unsupervised pre-training typically involves replacing multiple words with [MASK] based on a massive corpus of Chinese language data from the internet. The original text in the replaced words is then predicted based on the context. Language models trained through unsupervised pre-training typically possess strong word embedding and context modeling capabilities. Supervised fine-tuning primarily involves adding annotated data to fine-tune the model output based on the previous unsupervised pre-training phase, improving its usefulness and safety.

[0085] Reinforcement Learning from Human Feedback (RLHF) ranks and annotates model responses based on human feedback (e.g., selecting text descriptions based on human preferences). These annotated responses are then used to train a preference model, which returns a scalar reward to the reinforcement learning (RL) optimizer. Finally, a dialogue agent is trained through reinforcement learning to simulate the preference model. The output of a large language model can be input into a pre-trained preference model to obtain a score for the output. Based on the score, the parameters of the large language model are continuously adjusted to ensure that the output of the large language model is more consistent with human preferences and has more accurate grammar.

[0086] Instruction tuning is used to provide more explicit instructions to help large language models understand and output correct responses. Instruction tuning primarily uses instructions to guide the model to perform specific operations or complete tasks. Instructions can be computer programs or scripts, or user-written guidance text. The purpose of instructions is to tell the model how to process data or perform an operation, rather than simply providing context or task-related information.

[0087] For example, if the input is: A traffic congestion incident occurred at XX intersection on X month X day in 2023. What is the location of this incident? The label target is: Xizhimen intersection.

[0088] The instruction-tuned model will be better at performing downstream tasks similar to the training task without the need for separate training for each example.

[0089] Through a multi-layer training process, the understanding and generation capabilities of the large language model are improved, thereby improving the output accuracy of the large language model.

[0090] In one embodiment, the event processing method based on the large language model further includes: parsing the question to obtain key words; storing the key words and the reply content corresponding to the question in correspondence, so as to obtain and feedback the reply content corresponding to the key words when receiving the question corresponding to the key words.

[0091] Extract keywords from the question. Keywords are used to describe the question. Keywords and replies are stored in a corresponding manner, and replies can be retrieved based on the keywords. For example, keywords can be used as keys and replies as values, and both can be stored in a key-value format. When similar questions are subsequently received, the corresponding replies can be retrieved directly based on the keywords, speeding up responses.

[0092] By analyzing the question to obtain key words, and storing the key words and the corresponding reply content of the question in correspondence, redundant calculation operations can be reduced and the reply speed can be accelerated.

[0093] In addition, the large language model can also conduct question-and-answer sessions in the form of multi-round conversations, adding the content of the multi-round conversations to the standard prompt information, increasing the input content, and inputting it into the large language model to improve the input accuracy of the large language model and the accuracy of its output.

[0094] According to the technical solution of the present application, by filtering event information based on questions, target information is obtained, and the multiple target information obtained by filtering are integrated with the input questions as input data and input into the large language model. This can reduce redundant input data and improve data processing speed. At the same time, it can increase the contextual content related to the question, increase the representativeness of the input, and improve the output accuracy of the large language model.

[0095] Figure 4 is a scenario diagram of an event information generation method provided by an embodiment of the present application; Figure 5 is a scenario diagram of a training method based on a large language model provided by an embodiment of the present application; Figure 6 is a schematic diagram of a reply content provided by an embodiment of the present application. The embodiment of the present application is mainly divided into four stages: (1) open vocabulary target perception, (2) prompt vocabulary summary, (3) large language model and (4) question and answer analysis.

[0096] (1) The goal of open vocabulary object perception is to establish image and text matching capabilities. Based on the captured image and the text description of the scene event type to be perceived, the region corresponding to the text description is found in the image and the coordinates of the four points of the region's boundary are selected. Unlike traditional fixed-category object detection, open vocabulary object detection can accept any text description as input, thus expanding the number of object detection categories to nearly infinite.

[0097] As shown in Figure 4, BERT can be used as a text encoder to extract features from scene event text to obtain scene text features. A Vision Transformer can be used as an image encoder to extract features from scene images to obtain multiple scene image features. The scene text features and scene image features are input into Cross Modal Query Selection to obtain the similarity between the scene text features and each scene image feature. The 300 scene image features with the highest similarity are selected from the 900 scene image features, and each of these 300 scene image features is identified as a target image feature. The target image feature is input into a multimodal decoder to obtain candidate boxes, for example, the location of the candidate box. The coordinates of the four vertices of the rectangle representing the candidate box's location are typically used to represent the candidate box's location. For each candidate box, the similarity between the scene text feature and the candidate box is calculated and used as the candidate box's confidence. The similarity between the scene image feature that determined the candidate box and the scene text feature can be used as the similarity between the scene text feature and the candidate box. The target box is selected based on the candidate box's confidence.

[0098] (2) Prompt vocabulary total module

[0099] The open vocabulary-aware detection results (such as the target box position, acquisition time, acquisition device identifier, and acquisition device location) are converted into natural language form for large language model input. The original detection results are mainly composed of a series of key-value pairs, such as:

[0100] {

[0101] camera_id:123123123,

[0102] camera_name:XX intersection,

[0103] event: vehicle driving in the wrong direction,

[0104] bbox:[123,234,345,456],

[0105] time:XXX,

[0106] }

[0107] The prompt word template is used to assemble the keywords in the detection results into the natural language input. For example, the prompt word template corresponding to the above key-value pair can be:

[0108] "Event {event} occurred at location {camera_name} at {time}"

[0109] After each event occurs, the above template is filled in with natural language to obtain event information, which is temporarily stored for subsequent input into the large language model for aggregation.

[0110] (3) Large Language Model

[0111] The large language model uses a generative understanding model trained on Chinese corpus. The pre-training task of the large language model is divided into three stages:

[0112] 1. Unsupervised pre-training, as shown in Figure 5, typically involves replacing multiple words with [MASK] from a massive corpus of Chinese internet data. The original text of the words replaced by [MASK] is then predicted based on the context. Language models that have undergone unsupervised pre-training typically have relatively good word embedding and context modeling capabilities.

[0113] 2. Supervised fine-tuning

[0114] Supervised fine-tuning mainly adds labeled data to fine-tune the model based on the previous stage to improve the usefulness and harmlessness of the model output.

[0115] 3. RLHF

[0116] RLHF ranks and annotates the model's responses based on human feedback (e.g., selecting text descriptions based on human preferences). These annotated responses are then used to train a preference model, which returns a scalar reward to the RL optimizer. Finally, a dialogue agent is trained through reinforcement learning to simulate the preference model. The output of the large language model can be input into a pre-trained preference model to obtain a score for the output. Based on the score, the parameters of the large language model are continuously adjusted to ensure that the output of the large language model is more consistent with human-preferred expressions and more accurate grammar.

[0117] After the above process, a large language model that can handle general tasks is obtained, which then needs to be fine-tuned based on business objectives based on the Instruction Tuning method.

[0118] Instruction tuning primarily uses instructions to guide the model to perform specific operations or complete tasks. Instructions can be computer programs or scripts, or user-written guidance text. The purpose of instructions is to tell the model how to process data or perform an operation, rather than simply providing context or task-related information.

[0119] For example, if the input is: A traffic congestion incident occurred at XX intersection on X month X day in 2023. What is the location of this incident? The label target is: Xizhimen intersection.

[0120] (4) Question and Answer Analysis

[0121] Question and answer parsing can provide an interface for interacting with users. Its main function is to input the output structured results (event information) and other queries entered by users into the large language model and parse the results.

[0122] This solution presets a series of query templates for periodically outputting default content.

[0123] For example, you are the mayor's assistant. The following are unusual events that occurred in the city today: {Event Information}. Please summarize today's high-incidence and unusual events and form a report. The response content can be referred to as Figure 6.

[0124] In addition, this module receives users' open queries and inputs them into a large language model for question and answer in the form of multi-round dialogues, which can support users' open needs.

[0125] The model output is structured and parsed based on pre-set code, extracting keywords and storing them as key-value pairs. When a user enters a query, the keyword is filtered and matched with the stored key. If a match is successful, the corresponding value is output.

[0126] This application automatically senses and aggregates unusual events within urban scenarios, enabling conversations based on relevant content. Compared to traditional approaches that sense events and display them within business systems, this embodiment can efficiently extract open questions such as "Find the most congested road sections" and "Has urban flooding improved in August compared to July?"

[0127] According to an embodiment of the present application, Figure 7 is a structural diagram of an event processing device based on a large language model, provided in an embodiment of the present application. This embodiment of the present application is applicable to situations where a large language model responds to collected events. The device is implemented using software and / or hardware and is configured in an electronic device with sufficient data processing capabilities.

[0128] As shown in FIG7 , an event processing device 700 based on a large language model includes: an input question acquisition module 701, an event information acquisition module 702, an event question fusion module 703, and a reply processing module 704. Specifically, the input question acquisition module 701 is configured to acquire a question input by a user; the event information acquisition module 702 is configured to acquire pre-generated event information; the event question fusion module 703 is configured to fuse the question with the event information to obtain a fusion result; and the reply processing module 704 is configured to input the fusion result into a pre-trained large language model to obtain a reply corresponding to the question.

[0129] According to the technical solution of the present application, by obtaining pre-generated event information and fusing it with the question input by the user, a fusion result is obtained, and the fusion result is input into a large language model for question understanding and generation, and the reply content corresponding to the question is obtained. The context associated with the question can be increased, and events can be aggregated. Question replies are given based on the aggregated events, which can improve the prediction accuracy of the reply content, especially for application scenarios with question texts, and can improve the processing accuracy of the reply text.

[0130] FIG8 is a structural diagram of an event information acquisition module provided by an embodiment of the present application. The event information acquisition module 702 includes: a scene event type acquisition unit 801, configured to acquire a pre-set scene event type; a scene image screening unit 802, configured to screen scene images corresponding to the scene event type from a plurality of acquired images; a target frame detection unit 803, configured to detect a target frame corresponding to the scene event type from the scene image; an acquisition information acquisition unit 804, configured to acquire acquisition information of the scene image; an event information generation unit 805, configured to generate event information based on the scene event type, the acquisition information of the scene image, and the target frame; and an event information acquisition unit 806, configured to acquire pre-generated event information from a plurality of generated event information.

[0131] FIG9 is a structural diagram of a target frame detection unit provided in an embodiment of the present application. The target frame detection unit 803 includes: a text feature extraction subunit 901, configured to extract features from the text of the scene event type to obtain scene text features; an image feature extraction subunit 902, configured to extract features from the scene image to obtain multiple scene image features; an image feature screening subunit 903, configured to screen multiple target image features similar to the scene text features from the multiple scene image features; a candidate frame subunit 904, configured to detect candidate frames corresponding to the scene event type based on each target image feature; a similarity calculation subunit 905, configured to calculate the similarity between each candidate frame corresponding to the scene event type and the scene text features; a confidence calculation subunit 906, configured to calculate the confidence of each candidate frame corresponding to the scene event type based on the similarity between each candidate frame corresponding to the scene event type and the scene text features; and a target frame detection subunit 907, configured to determine the target frame corresponding to the scene event type from the multiple candidate frames corresponding to the scene event type based on the confidence of each candidate frame corresponding to the scene event type.

[0132] Figure 10 is a structural diagram of an event-question fusion module provided in an embodiment of the present application. The number of pre-generated event information is multiple, and the event-question fusion module 703 includes: an event information screening unit 1001, configured to screen the multiple acquired event information to obtain target information corresponding to the question; a target information fusion unit 1002, configured to fuse the question with the target information; and the number of target information is at least one.

[0133] Figure 11 is a structural diagram of another event-question fusion module provided in an embodiment of the present application. The event-question fusion module 703 includes: a prompt information acquisition unit configured to acquire standard prompt information 1003, wherein the standard prompt information includes a slot for the event information and a slot for a question; an event information fusion unit 1004 configured to fill the event information into the slot for the event information; and a question fusion unit 1005 configured to fill the question into the slot for the question to obtain a fusion result.

[0134] FIG12 is a block diagram of another large language model-based event processing device provided in an embodiment of the present application. The large language model-based event processing device further includes: a real-time video acquisition module 705 configured to acquire a real-time video stream and acquisition information of the real-time video stream; an image acquisition module 706 configured to acquire multiple acquired images from the real-time video stream; and an acquisition information acquisition module 707 configured to determine acquisition information for each acquired image based on the acquisition information of the real-time video stream; wherein the acquisition information includes an acquisition device identifier, an acquisition device location, and an acquisition time.

[0135] In one embodiment, the collection device includes a traffic monitoring camera within a city; the collection device location includes a shooting location of the traffic monitoring camera.

[0136] In one embodiment, the pre-trained large language model is determined by instruction tuning of a large language model for processing general tasks, and the large language model for processing general tasks is obtained through unsupervised training, labeled supervised fine-tuning, and human feedback reinforcement learning training.

[0137] Figure 13 is a block diagram of another event processing device based on a large language model provided by an embodiment of the present application. The event processing device based on a large language model further includes: a keyword extraction module 708, configured to parse the question to obtain keywords; and a reply storage module 709, configured to store the keywords and the replies corresponding to the questions in a corresponding manner, so as to obtain and feedback the replies corresponding to the keywords when a question corresponding to the keywords is received.

[0138] The above-mentioned event processing device based on the large language model can execute the event processing method based on the large language model provided in any embodiment of the present application, and has the corresponding functional modules and effects for executing the event processing method based on the large language model.

[0139] In the technical solution of this application, the collection, storage, use, processing, transmission, provision and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0140] According to an embodiment of the present application, the present application also provides an electronic device, a readable storage medium and a computer program product.

[0141] Figure 14 shows a schematic area diagram of an example electronic device 800 that can be used to implement an embodiment of the present application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0142] As shown in FIG14 , the device 800 includes a computing unit 801 that can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. The RAM 803 can also store various programs and data required for the instructions of the device 800. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0143] Various components in device 800 are connected to I / O interface 805, including an input unit 806, such as a keyboard, mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a magnetic disk, optical disk, etc.; and a communication unit 809, such as a network card, modem, wireless communication transceiver, etc. The communication unit 809 allows device 800 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0144] The computing unit 801 can be a variety of general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of the computing unit 801 include a central processing unit (CPU), a graphics processing unit (GPU), a variety of dedicated artificial intelligence (AI) computing chips, a variety of computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 801 performs the multiple methods and processes described above, such as the event processing method based on the large language model. For example, in some embodiments, the event processing method based on the large language model can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as the storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the event processing method based on the large language model described above can be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to execute the event processing method based on the large language model in any other appropriate manner (for example, by means of firmware).

[0145] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0146] The program code for implementing the method of the present application can be written in any combination of one or more programming languages. Such program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / instructions specified in the flow chart and / or area diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0147] In the context of the present application, machine-readable medium can be a tangible medium that can contain or store a program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable medium can include electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, RAM, ROM, erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0148] To provide for user interaction, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD) monitor) configured to display information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be configured to provide for user interaction; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic input, voice input, or tactile input.

[0149] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: Local Area Networks (LANs), Wide Area Networks (WANs), blockchain networks, and the Internet.

[0150] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. This client-server relationship is established by computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host, a host product within a cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosts and virtual private servers (VPS) services. The server may also be a server in a distributed system or a server integrated with blockchain.

[0151] Artificial intelligence (AI) is the study of how computers can simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, and big data processing. AI software technologies primarily encompass computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graphs.

[0152] Cloud computing refers to a technology system that provides network access to elastically scalable shared pools of physical or virtual resources. These resources can include servers, instruction sets, networks, software, applications, and storage devices, and can be deployed and managed on-demand in a self-service manner. Cloud computing technology provides efficient and powerful data processing capabilities for the application of technologies such as artificial intelligence and blockchain, as well as for model training.

[0153] The various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the multiple steps described in this application can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution provided by this application can be achieved. This document is not limited here.

Claims

1. An event processing method based on a large language model, comprising: Get user input questions; Get pre-generated event information; Fusing the problem and the event information to obtain a fusion result; The fusion result is input into a pre-trained large language model to obtain a response content corresponding to the question.

2. The method according to claim 1, before obtaining the pre-generated event information, further comprising: Get the preset scene event type; Selecting a scene image corresponding to the scene event type from among the collected multiple images; In the scene image, detecting a target frame corresponding to the scene event type; Acquiring acquisition information of the scene image; Generate event information according to the scene event type, the acquisition information of the scene image, and the target frame; The obtaining of pre-generated event information includes: The pre-generated event information is acquired from the generated plurality of event information.

3. The method according to claim 2, wherein: The step of detecting a target frame corresponding to the scene event type in the scene image includes: Extracting features from the text of the scene event type to obtain scene text features; Extracting features from the scene image to obtain multiple scene image features; Selecting a plurality of target image features similar to the scene text features from among the plurality of scene image features; According to each target image feature, detecting a candidate box corresponding to the scene event type; Calculating the similarity between each candidate box corresponding to the scene event type and the scene text feature; Calculating the confidence of each candidate box corresponding to the scene event type according to the similarity between each candidate box corresponding to the scene event type and the scene text feature; According to the confidence of each candidate box corresponding to the scene event type, a target box corresponding to the scene event type is determined from the multiple candidate boxes corresponding to the scene event type.

4. The method according to claim 1, wherein: The number of the pre-generated event information is multiple, and the merging of the question and the event information includes: Screening the acquired multiple event information to obtain target information corresponding to the problem, wherein the number of the target information is at least one; The question is fused with the target information.

5. The method according to claim 1, wherein: The step of fusing the problem information with the event information to obtain a fusion result includes: Obtaining standard prompt information, wherein the standard prompt information includes a slot of the event information and a question slot; Fill the event information into the slot of the event information; Fill the question into the question slot to obtain the fusion result.

6. The method according to claim 2, before acquiring the acquisition information of the scene image, further comprises: Acquire a real-time video stream and acquisition information of the real-time video stream; Acquire a plurality of collected images from the real-time video stream; According to the acquisition information of the real-time video stream, the acquisition information of each acquired image is determined; wherein the acquisition information includes an acquisition device identifier, an acquisition device location and an acquisition time.

7. The method according to claim 6, wherein: The collection equipment includes traffic monitoring cameras within the city; the collection equipment location includes the shooting location of the traffic monitoring camera.

8. The method according to claim 1, wherein: The pre-trained large language model is determined by tuning a large language model for processing general tasks through instructions, and the large language model for processing general tasks is obtained through unsupervised training, labeled supervised fine-tuning, and human feedback reinforcement learning training.

9. The method according to claim 1, further comprising: Analyze the problem and obtain key words; The key words and the reply contents corresponding to the questions are stored correspondingly, and in response to receiving the questions corresponding to the key words, the reply contents corresponding to the key words are acquired and fed back.

10. An event processing device based on a large language model, comprising: An input question acquisition module is configured to acquire questions input by users; An event information acquisition module, configured to acquire pre-generated event information; An event question fusion module, configured to fuse the question and the event information to obtain a fusion result; The reply processing module is configured to input the fusion result into a pre-trained large language model to obtain reply content corresponding to the question.

11. The device according to claim 10, wherein: The event information acquisition module includes: A scene event type acquisition unit, configured to acquire a preset scene event type; A scene image screening unit, configured to screen a scene image corresponding to the scene event type from among the multiple images collected; a target frame detection unit, configured to detect a target frame corresponding to the scene event type in the scene image; An acquisition information acquisition unit, configured to acquire acquisition information of the scene image; an event information generating unit, configured to generate event information according to the scene event type, the acquisition information of the scene image and the target frame; The event information acquisition unit is configured to acquire pre-generated event information from among the generated plurality of event information.

12. The device according to claim 11, wherein The target frame detection unit includes: A text feature extraction subunit, configured to extract features from the text of the scene event type to obtain scene text features; An image feature extraction subunit, configured to extract features from the scene image to obtain a plurality of scene image features; An image feature screening subunit, configured to screen a plurality of target image features similar to the scene text features from the plurality of scene image features; A candidate box subunit is configured to detect a candidate box corresponding to the scene event type according to each target image feature; A similarity calculation subunit, configured to calculate the similarity between each candidate box corresponding to the scene event type and the scene text feature; a confidence calculation subunit, configured to calculate the confidence of each candidate box corresponding to the scene event type according to the similarity between each candidate box corresponding to the scene event type and the scene text feature; The target frame detection subunit is configured to determine a target frame corresponding to the scene event type from among multiple candidate frames corresponding to the scene event type according to the confidence of each candidate frame corresponding to the scene event type.

13. The device according to claim 10, wherein: The number of the pre-generated event information is multiple, and the event problem fusion module includes: an event information screening unit, configured to screen the acquired multiple event information to obtain target information corresponding to the problem, wherein the number of the target information is at least one; The target information fusion unit is configured to fuse the question with the target information.

14. The device according to claim 10, wherein: The event problem fusion module includes: A prompt information acquisition unit, configured to acquire standard prompt information, wherein the standard prompt information includes a slot of the event information and a question slot; An event information fusion unit, configured to fill the event information into the slot of the event information; The question fusion unit is configured to fill the question into the question slot to obtain the fusion result.

15. The apparatus according to claim 11, further comprising: A real-time video acquisition module, configured to acquire a real-time video stream and acquisition information of the real-time video stream; An image acquisition module, configured to acquire a plurality of collected images from the real-time video stream; The acquisition information acquisition module is configured to determine the acquisition information of each acquired image according to the acquisition information of the real-time video stream; wherein the acquisition information includes the acquisition device identification, the acquisition device location and the acquisition time.

16. The device according to claim 15, wherein: The collection equipment includes traffic monitoring cameras within the city; the collection equipment location includes the shooting location of the traffic monitoring camera.

17. The device according to claim 10, wherein: The pre-trained large language model is determined by tuning a large language model for processing general tasks through instructions, and the large language model for processing general tasks is obtained through unsupervised training, labeled supervised fine-tuning, and human feedback reinforcement learning training.

18. The apparatus according to claim 10, further comprising: A keyword extraction module is configured to parse the question to obtain keywords; The reply storage module is configured to store the reply contents corresponding to the keywords and the questions in correspondence, and in response to receiving the questions corresponding to the keywords, obtain and feed back the reply contents corresponding to the keywords.

19. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform any one of claims 1 to 9. The event processing method based on the large language model.

20. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable the computer to execute the event processing method based on a large language model according to any one of claims 1-9.

21. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the event processing method based on a large language model according to any one of claims 1 to 9.

Images