Event processing method, device, equipment and medium based on large language model
By integrating user-input problems and event information in large language model, more accurate reply content is generated, and the problem of lack of context and accuracy of reply content in event processing in the prior art is solved, and the efficiency and accuracy of urban event processing is improved. sex.
Patent Information
- Application Number
- CN202311416014.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-10-27
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2043-10-27
AI Technical Summary
The prior art is difficult to effectively integrate user-input problems and pre-generated event information in urban event processing, resulting in a lack of context and accuracy in reply content, which affects the efficiency and accuracy of problem processing.
The event processing method based on the large language model is adopted, and the problem input by obtaining user-entered and pre-generated event information is fused, and input it into the pre-trained large language model to generate more accurate reply content. The method includes obtaining questions entered by the user, obtaining pre-generated event information, fusing the two into a large language model, and generating replies corresponding to the input questions.
It improves the speed and accuracy of problem responses, enhances the context relevance of reply content and real-time event processing, especially in urban event processing scenarios, reduces the difficulty of searching and filtering, and improves the accuracy of reply.
Smart Images

Figure CN117407507B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of large models, specifically to the fields of artificial intelligence, large language models and human-computer interaction, and in particular to an event processing method, device, equipment and medium based on a large language model. Background Art
[0002] In recent years, intelligent systems have improved the level of refined and intelligent urban governance through full perception, full interconnection, full analysis, full response and full application of urban operation status.
[0003] Intelligent systems can connect the core elements of a city, achieve efficient allocation of public resources, and accurately handle urban events. Summary of the invention
[0004] The present disclosure provides an event processing method, apparatus, device and medium based on a large language model.
[0005] According to one aspect of the present disclosure, there is provided an event processing method based on a large language model, comprising:
[0006] Get user input questions;
[0007] Get pre-generated event information;
[0008] Fusing the input question and the event information to obtain a fusion result;
[0009] The fusion result is input into a pre-trained large language model to obtain a reply content corresponding to the input question.
[0010] According to one aspect of the present disclosure, there is provided an event processing device based on a large language model, comprising:
[0011] Input question acquisition module, used to obtain questions input by users;
[0012] An event information acquisition module, used to acquire pre-generated event information;
[0013] An event question fusion module, used to fuse the input question with the event information to obtain a fusion result;
[0014] The reply processing module is used to input the fusion result into a pre-trained large language model to obtain reply content corresponding to the input question.
[0015] According to another aspect of the present disclosure, there is provided an electronic device, comprising:
[0016] at least one processor; and
[0017] a memory communicatively connected to the at least one processor; wherein,
[0018] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the event processing method based on the large language model described in any embodiment of the present disclosure.
[0019] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the event processing method based on a large language model described in any embodiment of the present disclosure.
[0020] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the event processing method based on a large language model described in any embodiment of the present disclosure is implemented.
[0021] The disclosed embodiments can improve the speed of answering questions.
[0022] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.
[0024] Figure 1 is a flow chart of an event processing method based on a large language model according to an embodiment of the present disclosure;
[0025] Figure 2 is a flowchart of another event processing method based on a large language model disclosed in an embodiment of the present disclosure;
[0026] Figure 3 is a flowchart of another event processing method based on a large language model disclosed in an embodiment of the present disclosure;
[0027] Figure 4 It is a scene diagram of a method for generating event information according to an embodiment of the present disclosure;
[0028] Figure 5 It is a scene diagram of a training method based on a large language model disclosed in an embodiment of the present disclosure;
[0029] Figure 6 is a schematic diagram of a reply content disclosed according to an embodiment of the present disclosure;
[0030] Figure 7 is a structural diagram of an event processing device based on a large language model disclosed in an embodiment of the present disclosure;
[0031] Figure 8 It is a block diagram of an electronic device according to the event processing method based on a large language model disclosed in an embodiment of the present disclosure. DETAILED DESCRIPTION
[0032] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.
[0033] Figure 1 This is a flowchart of an event processing method based on a large language model according to an embodiment of the present disclosure. This embodiment can be applied to the case where a large language model responds based on collected events. The method of this embodiment can be executed by an event processing device based on a large language model, which can be implemented in software and / or hardware, and is specifically configured in an electronic device with a certain data computing capability, which can be a client device or a server device, such as a mobile phone, a tablet computer, a vehicle-mounted terminal, and a desktop computer.
[0034] S101, obtaining a question input by a user.
[0035] The question may refer to the natural language content input by the user during the human-computer interaction process. The user may input data in at least one type of form such as text, image, voice or video, and the directly input data may be recognized to obtain the question text. The client may receive the user input data and send it to the electronic device of the embodiment of the present disclosure, and the electronic device obtains the question text according to the input data. Optionally, the question input by the user is the question text.
[0036] S102: Obtain pre-generated event information.
[0037] Event information may refer to information collected about real events that already exist or have occurred. Events may refer to events that occur within a certain area. For example, events that occur in a certain city. For example, events may be abnormal events or normal events. For another example, events may be traffic events, environmental events, business events, or urban management events. Events may be specific events in a certain field or comprehensive events in multiple fields.
[0038] Events can be collected and summarized. The number of collected event information is at least one. Events can be data perceived in the real world. Subsequent processing based on the perceived event information can improve the real-time nature of event processing. It should be noted that the collection, processing and display of events are all authorized in a legal and compliant manner.
[0039] S103: Fusing the input question and the event information to obtain a fusion result.
[0040] The fusion result is used as the final input to the large language model. The fusion result includes question and event information, where the included event information is used to enrich the context of the available question and can serve as supplementary content for the question.
[0041] S104: Input the fusion result into a pre-trained large language model to obtain a reply content corresponding to the input question.
[0042] The large language model is used to understand natural language content and generate corresponding reply content. Optionally, the large language model is a question-answering model, which is used to answer the question content and generate reply content to feed back to the user. The question and event information are input into the large language model together, so that the large language model can understand the question and the event information in the context, and can understand more accurate content from the event information, thereby generating more accurate reply content.
[0043] In a specific example, the event information includes: the traffic congestion event in city A on that day, and the question is: the traffic handling strategy of city A on that day. The reply content is: the traffic resource allocation strategy of the congested road section in city A at a certain time on that day, and the traffic resource allocation strategy of the remaining road sections in the remaining time.
[0044] According to the technical solution of the present invention, by obtaining pre-generated event information and fusing it with the question input by the user to obtain a fusion result, the fusion result is input into a large language model for question understanding and generation, and the reply content corresponding to the question is obtained. The context associated with the question can be increased, and events can be summarized. Question replies are given based on the summarized events, which can improve the prediction accuracy of the reply content, especially for application scenarios with question texts, and can improve the processing accuracy of the reply text.
[0045] Figure 2It is a flowchart of another event processing method based on a large language model disclosed in an embodiment of the present disclosure, which is further optimized and expanded based on the above technical solution and can be combined with the above optional implementation methods. The first intermediate state includes at least one of the following: an action, an action description content, an action reasoning process, an intermediate result, and an action identifier; wherein the action is obtained by screening the large language model from a plurality of preset actions.
[0046] S201: Obtain a preset scene event type.
[0047] The scene event type is used to distinguish the collected events. Usually, the scene event type can be represented by a keyword. The scene event can be a typical event, for example, an event that occurs during urban management or urban operation. For example, the scene event type is a traffic accident, another example, the scene event type is parking in a parking lot, and another example, the scene event type is waterlogging.
[0048] The scene event type can be pre-input by the user according to the problem to be solved. The first user can input the scene event type and the second user can input the problem. Or the same user can input the scene event type and the problem. The scene event type can usually be pre-defined based on the problem to be solved. For example, if the user wants to know the traffic conditions in the city, the scene event type can be set to traffic congestion, traffic restricted road section or traffic accident, etc., and the event information related to the scene event type can be collected and generated based on the data generated in real time in the city. So that questions can be asked based on the event information related to the scene event type.
[0049] S202: Filter the scene images corresponding to the scene event type from the multiple collected images.
[0050] Among the multiple images collected, there are images that are irrelevant to the scene event type, wherein the irrelevant images do not contain content related to the scene event type. For example, in the previous example, the scene event type is a traffic accident, and the collected images include images of small animals, images of two cars colliding, and park monitoring images, wherein the image of the two cars colliding is the scene image corresponding to the traffic accident, and the other images are images that are irrelevant to the traffic accident.
[0051] The description content of multiple images can be identified to determine the description information corresponding to each image, and the images with the same or similar description information as the scene event type can be determined as scene images. For another example, the text of the scene event type and each image can be input into a pre-trained deep learning model for content matching to obtain the scene image corresponding to the scene event type. In addition, there are other situations, which are not specifically limited.
[0052] Optionally, the event processing method based on the large language model also includes: obtaining a real-time video stream and acquisition information of the real-time video stream; obtaining multiple acquired images from the real-time video stream; determining acquisition information of each of the acquired images based on the acquisition information of the real-time video stream; wherein the acquisition information includes an acquisition device identifier, an acquisition device location, and an acquisition time.
[0053] A real-time video stream may refer to a video stream collected in real time. The real-time video stream is intercepted or sampled to obtain a plurality of collected images. The image collection time is determined according to the collection information of the real-time video stream. The collection information of the real-time video stream may include the identification of the collection device that collects the real-time video stream, the location of the collection device, and the collection time, wherein the identification of the image and the real-time video stream collection device is the same, and the collection device location is the same. The real-time video stream records the time of recording each frame, and the image collection time can be determined according to the order or time of the image in the real-time video stream.
[0054] By acquiring the collected image from the real-time video stream and determining the collection information of the collected image according to the collection information of the real-time video stream, the real-time nature of the event can be improved, thereby improving the event perception speed.
[0055] Optionally, the collection device includes a traffic monitoring camera within a city; the collection device location includes a shooting location of the traffic monitoring camera.
[0056] Traffic monitoring cameras are used to monitor road anomalies. The location of the acquisition device may include the location and / or shooting location of the traffic monitoring camera. Exemplarily, the location of the acquisition device may be intersection A. In a specific example, the problem may be to find the road section with the highest traffic congestion.
[0057] By setting the application scenario to traffic monitoring events, traffic events within urban scenes can be summarized, and questions about traffic monitoring events can be answered on this basis. Real-time urban traffic events can be extracted and summarized, reducing the difficulty of searching and screening urban traffic events and improving the accuracy of answering questions about urban traffic events.
[0058] S203: Detect a target frame corresponding to the scene event type in the scene image.
[0059] The target frame may refer to an image content area corresponding to the scene event type. The image content in the target frame is the same or similar to the scene event type. For example, if the scene event type is uncivilized pet keeping, the image in the target frame may be an image of an untied pet dog on the road. The scene image is an image related to the scene event type obtained by screening, and the presence of a target frame in the scene image corresponds to the scene event type.
[0060] Optionally, detecting a target box corresponding to the scene event type in the scene image includes: performing feature extraction on the text of the scene event type to obtain scene text features; performing feature extraction on the scene image to obtain multiple scene image features; screening target image features similar to the scene text features among the multiple scene image features; detecting an alternative box corresponding to the scene event type based on the target image features; calculating the similarity between the alternative box corresponding to the scene event type and the scene text features; calculating the confidence of the alternative box corresponding to the scene event type based on the similarity between the alternative box corresponding to the scene event type and the scene text features; and determining, based on the confidence of the alternative box corresponding to the scene event type, a target box corresponding to the scene event type in the alternative box corresponding to the scene event type.
[0061] The text of the scene event type may refer to the content of the text being the scene event type. Feature extraction is performed on the text of the scene event type to obtain scene text features. For example, BERT (Bidirectional Encoder Representation from Transformers) may be used as a text encoder to extract features from the text of the scene event type to obtain scene text features. Feature extraction is performed on the scene image to obtain multiple scene image features. For example, Vision Transformer (visual transformer model) may be used as an image encoder to extract features from the scene image to obtain multiple scene image features.
[0062] The similarity between the scene text feature and the scene image feature is used to indicate the similarity between the content represented by the text feature and the content of the image feature. The scene text feature and the scene image feature can be input into the Cross Modal Query Selection (Cross Modal Query Selection Network) to obtain the similarity between the scene text feature and each scene image feature. According to the similarity, multiple scene image features with the highest similarity are screened and determined as the target image feature. For example, among the 900 scene image features, the 300 scene image features with the highest similarity are screened and determined as the target image feature. The target image feature is input into the decoder to obtain the candidate box, specifically the position of the candidate box, and the coordinates of the four vertices of the rectangle of the candidate box are usually used to represent the position of the candidate box. For each candidate box, the similarity between the scene text feature and the candidate box is calculated, and used as the confidence of the candidate box. Exemplarily, the similarity between the scene image feature and the scene text feature that determines the candidate box can be used as the similarity between the scene text feature and the candidate box.
[0063] By establishing the matching capability between images and texts for pre-set texts of any scene type, open-vocabulary target perception is achieved. According to the input image and the text description to be perceived, the area corresponding to the text description is found in the image, and the coordinates of the four points of the boundary of the area are selected to achieve open-vocabulary target detection. Therefore, any text description can be accepted as input, and the number of target detection categories can be expanded to nearly infinity, enriching the content of scene types and increasing application scenarios.
[0064] S204: Acquire acquisition information of the scene image.
[0065] The acquisition information of the scene image may include at least one of the following: the time, location, method, shooting content and shooting range of the scene image. When acquiring multiple acquired images, the acquisition information of each image can be acquired at the same time. Usually, the acquisition information of the image is recorded when the image is acquired.
[0066] S205: Generate event information according to the scene event type, the acquisition information of the scene image and the target frame.
[0067] The generated event information belongs to the scene event type. The collected information of the scene image is used to determine the content, state, and process of the event represented by the event information. The target frame is used to determine the basis of the event information, that is, to indicate that the event information really exists. By determining the event information through the scene event type, collected information, and target frame, the information related to the event can be recorded from multiple dimensions to enrich the content of the event. It can also be traced according to the target frame to facilitate the location of the event process and location.
[0068] S206: Obtain the question input by the user.
[0069] S207: Obtain pre-generated event information from the generated multiple event information.
[0070] Generate multiple event information in advance. For multiple event information, all generated event information can be obtained, or partially generated event information can be filtered, wherein the filtering condition can be time or similarity. If no event information is generated, the pre-generated event information can be empty. Among them, the order of S201-S205 and S206 can be swapped. Usually, after S201-S205 are executed, S206 is executed.
[0071] S208: Fusing the input question and the event information to obtain a fusion result.
[0072] Optionally, the input question and the event information are fused to obtain a fusion result, including: obtaining standard prompt information, the standard prompt information including a slot for the event information and a question slot; filling the event information into the slot for the event information; filling the question into the question slot to obtain a fusion result.
[0073] The standard prompt is a text prompt input to the model, which contains the keywords and context of the information or question that the user wants to query, so that the model can better understand the user's intention and give a more accurate response. The standard prompt is used to add event information and questions to form the input data of the large language model. The slot of event information is used to fill in event information, and the slot of question is used to fill in question. The fusion result can refer to the text obtained after the standard prompt is filled with event information and question.
[0074] In a specific example, the standard prompt message may be: You are now the mayor's assistant, and the following are abnormal events that occurred in the city today: {event information}, please handle {problem}.
[0075] By adapting standard prompt information to different questions and event information, the large language model can quickly understand questions and event information based on the prompt template and generate corresponding replies to the questions. The training step of the large language model understanding the semantics of different scenarios for different questions and generating corresponding replies is omitted, and the tuning process of the large language model is quickly implemented, reducing the labor cost of tuning and updating the large language model. At the same time, the corresponding prompt template can be quickly obtained according to the question in real time, improving the real-time update of the large language model and increasing the universality and versatility between the large language model and the question.
[0076] S209: Input the fusion result into a pre-trained large language model to obtain a reply content corresponding to the input question.
[0077] According to the technical solution disclosed in the present invention, image events are acquired by collecting multiple images, searching for scene images corresponding to scene event types, and detecting target frames corresponding to scene event types in scene images. Event information is generated based on the acquisition information of the scene images, the scene event types and the target frames. Events can be generated in real time, thereby improving the efficiency and authenticity of event acquisition.
[0078] Figure 3It is a flowchart of another event processing method based on a large language model disclosed in an embodiment of the present disclosure, which is further optimized and expanded based on the above technical solution and can be combined with the above optional implementation methods. The fusion of the input question and the event information is specifically as follows: screening the obtained multiple event information to obtain target information corresponding to the question; fusion of the input question and the target information; the number of the target information is at least one.
[0079] S301: Obtain a question input by a user.
[0080] S302: Obtain pre-generated event information.
[0081] S303: Filter the acquired multiple event information to obtain target information corresponding to the problem.
[0082] The target information is event information related to the question. For example, the question is about flooding, and the event information is about uncivilized pet keeping. The event information is irrelevant to the question. Multiple event information can be obtained, and event information irrelevant to the question may exist in the multiple event information. The event information can be filtered to retain only the content related to the target information, thereby reducing redundant input data.
[0083] S304: Fuse the input question with the target information to obtain a fusion result; the number of the target information is at least one.
[0084] There can be multiple target information obtained by screening. All target information can be integrated with the input question to enrich the input data.
[0085] Specifically, the standard prompt information is used to add the target information and questions to form the input data of the large language model. The slot of the event information is used to fill the target information, and the question slot is used to fill the question. The fusion result can refer to the text obtained after the standard prompt information is filled with the target information and the question.
[0086] As in the previous example, the standard prompt message may be: You are now the mayor's assistant, and the following are the abnormal events that occurred in the city today: {target information}, please handle {problem}.
[0087] S305: Input the fusion result into a pre-trained large language model to obtain a reply content corresponding to the input question.
[0088] Optionally, the pre-trained large language model is determined by instruction tuning a large language model for processing general tasks, and the large language model for processing general tasks is obtained through unsupervised training, labeled supervised fine-tuning, and human feedback reinforcement learning training.
[0089] The large language model for processing general tasks can be a large language model that does not distinguish between specific application fields and specific business scenario tasks and can achieve general understanding and generation functions. The large language model for processing general tasks can be a pre-trained model, and can also be fine-tuned to better adapt to specific application scenarios.
[0090] For a large language model that can handle general tasks, unsupervised pre-training is usually based on a large amount of Internet Chinese corpus, where some words are replaced with [MASK], and then the original text of the position removed by [MASK] is predicted based on the context. Language models that have undergone unsupervised pre-training usually have better WordEmbedding and context modeling capabilities. Supervised fine-tuning mainly adds annotated data to fine-tune the previous unsupervised pre-training stage to improve the usefulness and harmlessness of the model output.
[0091] Reinforcement Learning from Human Feedback (RLHF) is to sort and annotate the model's responses based on human feedback (e.g., selecting text descriptions based on human preferences). These annotated responses are then used to train a preference model, which is used to return a scalar reward for the RL (reinforcement learning) optimizer. Finally, the dialogue agent is trained through reinforcement learning to simulate the preference model. The output of the large language model can be input into a pre-trained preference model to obtain a score for the output result, and the parameters of the large language model can be continuously adjusted based on the score to make the output of the large language model more in line with human preferred expressions and more accurate grammar, etc.
[0092] Instruction tuning is used to provide more explicit instructions so that the large language model can understand and output the correct response content. Instruction tuning mainly guides the model to perform specific operations or complete tasks in the form of instructions. Instructions can be computer programs or scripts, or they can be guiding texts written by users. The purpose of instructions is to tell the model how to process data or perform an operation, rather than simply providing context or task-related information.
[0093] For example, input: A traffic congestion incident occurred at XX intersection on X month X day, 2023. What is the location of this incident? Tag Target: Xizhimen intersection.
[0094] The instruction-tuned model will be better at performing downstream tasks similar to the training task without the need to train each example separately.
[0095] Through a multi-layer training process, the understanding and generation capabilities of the large language model are improved, thereby improving the output accuracy of the large language model.
[0096] Optionally, the event processing method based on the large language model also includes: parsing the question to obtain key words; storing the key words and the reply content corresponding to the question in correspondence, so that when other questions corresponding to the key words are received, the reply content corresponding to the key words can be obtained for feedback.
[0097] Extract keywords from the question. Keywords are used to describe the content of the question. The keywords and the reply content are stored in correspondence, and the reply content can be queried based on the keywords. For example, the keywords can be used as keys and the reply content can be used as values, and the two can be stored in a key-value format. When similar questions are received later, the corresponding reply content can be directly queried based on the keywords, thereby improving the reply speed.
[0098] By analyzing the question to obtain key words, and storing the key words and the corresponding reply content of the question in correspondence, redundant calculation operations can be reduced, and the reply speed can be accelerated.
[0099] In addition, the large language model can also conduct question-and-answer sessions in the form of multi-round conversations, adding the content of the multi-round conversations to the standard prompt information, increasing the input content, and inputting it into the large language model to improve the accuracy of the input of the large language model and the accuracy of its output.
[0100] According to the technical solution disclosed in the present invention, by filtering event information based on questions to obtain target information, and fusing the filtered multiple target information with the input questions as input data and inputting them into a large language model, redundant input data can be reduced and data processing speed can be improved. At the same time, contextual content related to the questions can be increased, the representativeness of the input can be increased, and the output accuracy of the large language model can be improved.
[0101] Figure 4-Figure 6 This is a scene diagram of an event processing method based on a large language model disclosed in an embodiment of the present disclosure. The embodiment of the present disclosure is mainly divided into four stages: (1) open vocabulary target perception (2) prompt vocabulary summary (3) large language model (4) question and answer analysis.
[0102] (1) The purpose of open vocabulary object perception is to establish the matching ability between images and texts, find the area corresponding to the text description in the image based on the collected image and the text description of the scene event type to be perceived, and select the coordinates of the four points of the boundary of the area. Unlike traditional fixed-category object detection, open vocabulary object detection can accept any text description as input, so the number of object detection categories can be expanded to nearly infinity.
[0103] like Figure 4 As shown, BERT can be used as a text encoder to extract features from the text of the scene event type to obtain scene text features. Vision Transformer can be used as an image encoder to extract features from the scene image to obtain multiple scene image features. The scene text features and scene image features are input into Cross Modal Query Selection to obtain the similarity between the scene text features and each scene image feature. From the 900 scene image features, the 300 scene image features with the highest similarity are selected and determined as the target image features. The target image features are input into the multimodal decoder to obtain the candidate box, specifically the position of the candidate box, which is usually represented by the coordinates of the four vertices of the rectangle of the candidate box. For each candidate box, the similarity between the scene text feature and the candidate box is calculated and used as the confidence of the candidate box. The similarity between the scene image feature of the candidate box and the scene text feature can be used as the similarity between the scene text feature and the candidate box. According to the confidence of the candidate box, the target box is selected.
[0104] (2) Prompt vocabulary total module
[0105] The open vocabulary-aware detection results (the above-mentioned target box position, acquisition time, acquisition device identification and acquisition device location, etc.) are converted into natural language form for large language model input. The original detection results are mainly composed of a series of key-value pairs, such as:
[0106] {
[0107] camera_id:123123123,
[0108] camera_name:XX intersection,
[0109] event: vehicle driving in the wrong direction,
[0110] bbox:[123,234,345,456],
[0111] time:XXX,
[0112] }
[0113] The prompt word template is set to assemble the keywords in the detection results into the natural language input. For example, the prompt word template corresponding to the above key-value pair can be:
[0114] "Event {event} occurred at location {camera_name} at {time}"
[0115] After each event occurs, the natural language is filled in according to the above template to obtain event information, which is temporarily stored for subsequent input into the large language model for aggregation.
[0116] (3) Large Language Model
[0117] The large language model uses a comprehension generation model trained based on Chinese corpus. The pre-training task of the large language model is divided into three stages:
[0118] (1) Unsupervised pre-training, such as Figure 5 As shown in the figure, this stage is usually based on a large amount of Chinese corpus on the Internet, where some words are replaced with [MASK], and then the original text of the position replaced by [MASK] is predicted based on the context. The language model that has undergone unsupervised pre-training usually has better WordEmbedding and context modeling capabilities.
[0119] (2) Supervised fine-tuning
[0120] Supervised fine-tuning mainly adds labeled data to perform fine-tuning on the basis of the previous stage to improve the usefulness and harmlessness of the model output.
[0121] (3) Reinforcement Learning with Human Feedback (RLHF)
[0122] Reinforcement Learning from Human Feedback (RLHF) is to sort and annotate the model's responses based on human feedback (e.g., selecting text descriptions based on human preferences). These annotated responses are then used to train a preference model, which is used to return a scalar reward for the RL (reinforcement learning) optimizer. Finally, the dialogue agent is trained through reinforcement learning to simulate the preference model. The output of the large language model can be input into a pre-trained preference model to obtain a score for the output result, and the parameters of the large language model can be continuously adjusted based on the score to make the output of the large language model more in line with human preferred expressions and more accurate grammar, etc.
[0123] After the above process, a large language model that can handle general tasks is obtained, and then it needs to be fine-tuned based on business goals based on the Instruction Tuning method.
[0124] Instruction tuning mainly instructs the model to perform specific operations or complete tasks in the form of instructions. Instructions can be computer programs or scripts, or user-written guidance texts. The purpose of instructions is to tell the model how to process data or perform a certain operation, rather than simply providing context or task-related information.
[0125] For example, input: A traffic congestion incident occurred at XX intersection on X month X day, 2023. What is the location of this incident? Tag Target: Xizhimen intersection.
[0126] (4) Question and Answer Analysis
[0127] Question and answer parsing can provide an interface for interacting with users. Its main function is to input the output structured results (event information) and other queries entered by users into the large language model and parse the results.
[0128] This solution presets a series of query templates for periodically outputting default content.
[0129] For example, you are now the mayor's assistant. The following are the unusual events that occurred in the city today: {Event Information}. Please summarize today's high-incidence events and unusual events and form a report. The reply content can refer to the following: Figure 6 shown.
[0130] In addition, this module receives open-ended queries from users and inputs them into a large language model for question and answering in the form of multi-round dialogues, which can further support users' open-ended needs.
[0131] The model output results are structured and parsed based on the preset code, and keywords are extracted and stored as key-value pairs. When the user enters a query, the keyword is filtered and matched with the stored key. If the match is successful, the corresponding value is output.
[0132] The present disclosure automatically senses and summarizes abnormal events in urban scenes, and can conduct dialogues based on relevant content. Compared with the traditional solution of sensing events and displaying them in the business system, the present disclosure embodiment can efficiently extract some open questions, such as "find the road sections with the highest traffic congestion", "has the urban waterlogging phenomenon in August been improved compared to July", etc.
[0133] According to an embodiment of the present disclosure, Figure 7 1 is a structural diagram of an event processing device based on a large language model in an embodiment of the present disclosure. The embodiment of the present disclosure is applicable to the case where a large language model responds based on collected events. The device is implemented using software and / or hardware and is specifically configured in an electronic device with certain data computing capabilities.
[0134] like Figure 7 The event processing device 700 based on a large language model shown in the figure includes: an input question acquisition module 701, an event information acquisition module 702, an event question fusion module 703 and a reply processing module 704.
[0135] Input question acquisition module 701, used to acquire questions input by users;
[0136] An event information acquisition module 702 is used to acquire pre-generated event information;
[0137] An event question fusion module 703 is used to fuse the input question with the event information to obtain a fusion result;
[0138] The reply processing module 704 is used to input the fusion result into a pre-trained large language model to obtain reply content corresponding to the input question.
[0139] According to the technical solution of the present invention, by obtaining pre-generated event information and fusing it with the question input by the user to obtain a fusion result, the fusion result is input into a large language model for question understanding and generation, and the reply content corresponding to the question is obtained. The context associated with the question can be increased, and events can be summarized. Question replies are given based on the summarized events, which can improve the prediction accuracy of the reply content, especially for application scenarios with question texts, and can improve the processing accuracy of the reply text.
[0140] Furthermore, the event information acquisition module includes: a scene event type acquisition unit, used to acquire a preset scene event type; a scene image screening unit, used to screen scene images corresponding to the scene event type from multiple acquired images; a target frame detection unit, used to detect a target frame corresponding to the scene event type in the scene image; an acquisition information acquisition unit, used to acquire acquisition information of the scene image; an event information generation unit, used to generate event information based on the scene event type, the acquisition information of the scene image and the target frame; and an event information acquisition unit, used to acquire pre-generated event information from multiple generated event information.
[0141] Furthermore, the target frame detection unit includes: a text feature extraction subunit, which is used to extract features from the text of the scene event type to obtain scene text features; an image feature extraction subunit, which is used to extract features from the scene image to obtain multiple scene image features; an image feature screening subunit, which is used to screen target image features similar to the scene text features among the multiple scene image features; an alternative frame subunit, which is used to detect alternative frames corresponding to the scene event type based on the target image features; a similarity calculation subunit, which is used to calculate the similarity between the alternative frames corresponding to the scene event type and the scene text features; a confidence calculation subunit, which is used to calculate the confidence of the alternative frames corresponding to the scene event type based on the similarity between the alternative frames corresponding to the scene event type and the scene text features; and a target frame detection subunit, which is used to determine the target frame corresponding to the scene event type in the alternative frames corresponding to the scene event type based on the confidence of the alternative frames corresponding to the scene event type.
[0142] Furthermore, the event-problem fusion module 703 includes: an event information screening unit, used to screen the acquired multiple event information to obtain target information corresponding to the problem; a target information fusion unit, used to fuse the input problem with the target information; the number of the target information is at least one.
[0143] Furthermore, the event-problem fusion module 703 includes: a prompt information acquisition unit, used to acquire standard prompt information, wherein the standard prompt information includes a slot for the event information and a problem slot; an event information fusion unit, used to fill the event information into the slot for the event information; and a problem fusion unit, used to fill the problem into the problem slot to obtain a fusion result.
[0144] Furthermore, the event processing device based on the large language model also includes: a real-time video acquisition module, used to acquire real-time video streams and acquisition information of the real-time video streams; an image acquisition module, used to acquire multiple acquired images from the real-time video streams; an acquisition information acquisition module, used to determine the acquisition information of each of the acquired images based on the acquisition information of the real-time video streams; wherein the acquisition information includes an acquisition device identifier, an acquisition device location and an acquisition time.
[0145] Furthermore, the collection device includes a traffic monitoring camera within a city; the location of the collection device includes a shooting location of the traffic monitoring camera.
[0146] Furthermore, the pre-trained large language model is determined by tuning the large language model for processing general tasks through instructions, and the large language model for processing general tasks is obtained through unsupervised training, labeled supervised fine-tuning and human feedback reinforcement learning training.
[0147] Furthermore, the event processing device based on the large language model also includes: a keyword extraction module, used to parse the question to obtain keywords; a reply storage module, used to store the keywords and the reply content corresponding to the question in correspondence, so as to obtain the reply content corresponding to the keywords for feedback when other questions corresponding to the keywords are received.
[0148] The above-mentioned event processing device based on a large language model can execute the event processing method based on a large language model provided by any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects for executing the event processing method based on a large language model.
[0149] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0150] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.
[0151] Figure 8 A schematic area diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.
[0152] like Figure 8 As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the device 800 instructions can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.
[0153] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0154] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as an event processing method based on a large language model. For example, in some embodiments, the event processing method based on a large language model may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the event processing method based on the large language model described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to execute the event processing method based on the large language model in any other appropriate manner (for example, by means of firmware).
[0155] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard objects (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0156] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, implements the functions / instructions specified in the flow chart and / or the area diagram. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0157] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0158] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0159] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.
[0160] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services. The server may also be a server of a distributed system, or a server combined with a blockchain.
[0161] Artificial intelligence is a discipline that studies how to use computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, machine learning / deep learning technology, big data processing technology, knowledge graph technology, and other major directions.
[0162] Cloud computing refers to a technology system that uses network access to elastically scalable shared physical or virtual resource pools. Resources can include servers, instruction systems, networks, software, applications, and storage devices, and can be deployed and managed on demand and in a self-service manner. Cloud computing technology can provide efficient and powerful data processing capabilities for the application of technologies such as artificial intelligence and blockchain, as well as model training.
[0163] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution provided by this disclosure can be achieved, and this document does not limit this.
[0164] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.
Claims
1. An event processing method based on a large language model, include: Get user input questions; Get pre-generated event information; Fusing the input question and the event information to obtain a fusion result; Inputting the fusion result into a pre-trained large language model to obtain a response content corresponding to the input question; The obtaining of pre-generated event information includes: Obtaining a preset scenario event type, where the scenario event type is pre-input by the user according to a problem to be solved; Selecting a scene image corresponding to the scene event type from among the collected multiple images; In the scene image, detecting a target frame corresponding to the scene event type; Acquiring acquisition information of the scene image; Generate event information according to the scene event type, the acquisition information of the scene image, and the target frame; The pre-generated event information is acquired from among the generated plurality of event information.
2. The method according to claim 1, in, The step of detecting a target frame corresponding to the scene event type in the scene image includes: Extracting features from the text of the scene event type to obtain scene text features; Extracting features from the scene image to obtain multiple scene image features; Selecting a target image feature similar to the scene text feature from among the plurality of scene image features; Detecting a selection box corresponding to the scene event type according to the target image feature; Calculating the similarity between the candidate box corresponding to the scene event type and the scene text feature; Calculating the confidence of the candidate box corresponding to the scene event type according to the similarity between the candidate box corresponding to the scene event type and the scene text feature; According to the confidence of the candidate boxes corresponding to the scene event type, a target box corresponding to the scene event type is determined in the candidate boxes corresponding to the scene event type.
3. The method according to claim 1, in, The step of fusing the input question with the event information includes: Screening the acquired multiple event information to obtain target information corresponding to the problem; The input question is integrated with the target information; the number of the target information is at least one.
4. The method according to claim 1, in, The step of fusing the input question and the event information to obtain a fusion result includes: Obtaining standard prompt information, wherein the standard prompt information includes a slot of the event information and a question slot; Fill the event information into the slot of the event information; Fill the question into the question slot to obtain a fusion result.
5. The method according to claim 1, further comprising: include: Acquire a real-time video stream and acquisition information of the real-time video stream; Acquire a plurality of collected images from the real-time video stream; According to the acquisition information of the real-time video stream, the acquisition information of each of the acquired images is determined; wherein the acquisition information includes an acquisition device identifier, an acquisition device location and an acquisition time.
6. The method according to claim 5, in, The collection equipment includes traffic monitoring cameras within the city; the collection equipment location includes the shooting location of the traffic monitoring camera.
7. The method according to claim 1, in, The pre-trained large language model is determined by tuning a large language model for processing general tasks through instructions, and the large language model for processing general tasks is obtained through unsupervised training, labeled supervised fine-tuning, and human feedback reinforcement learning training.
8. The method according to claim 1, further comprising: include: Analyze the problem and obtain key words; The key words and the reply contents corresponding to the questions are stored correspondingly, so that when other questions corresponding to the key words are received, the reply contents corresponding to the key words are obtained for feedback.
9. An event processing device based on a large language model, include: Input question acquisition module, used to obtain questions input by users; An event information acquisition module, used to acquire pre-generated event information; An event question fusion module, used to fuse the input question with the event information to obtain a fusion result; A reply processing module, used for inputting the fusion result into a pre-trained large language model to obtain reply content corresponding to the input question; The event information acquisition module includes: A scene event type acquisition unit, used to acquire a preset scene event type, where the scene event type is pre-input according to the problem that the user needs to solve; A scene image screening unit, used to screen a scene image corresponding to the scene event type from among the multiple images collected; A target frame detection unit, configured to detect a target frame corresponding to the scene event type in the scene image; An acquisition information acquisition unit, used to acquire acquisition information of the scene image; An event information generating unit, configured to generate event information according to the scene event type, the acquisition information of the scene image and the target frame; The event information acquisition unit is used to acquire pre-generated event information from the generated multiple event information.
10. The device according to claim 9, in, The target frame detection unit includes: A text feature extraction subunit, used to extract features from the text of the scene event type to obtain scene text features; An image feature extraction subunit, used to extract features from the scene image to obtain a plurality of scene image features; An image feature screening subunit, configured to screen target image features similar to the scene text features from among the plurality of scene image features; A candidate box subunit, used to detect a candidate box corresponding to the scene event type according to the target image feature; A similarity calculation subunit, used to calculate the similarity between the candidate box corresponding to the scene event type and the scene text feature; A confidence calculation subunit, configured to calculate the confidence of the candidate box corresponding to the scene event type according to the similarity between the candidate box corresponding to the scene event type and the scene text feature; The target frame detection subunit is used to determine a target frame corresponding to the scene event type in the candidate frames corresponding to the scene event type according to the confidence of the candidate frames corresponding to the scene event type.
11. The device according to claim 9, in, The event problem fusion module includes: An event information screening unit, used to screen the acquired multiple event information to obtain target information corresponding to the problem; A target information fusion unit is used to fuse the input question with the target information; the number of the target information is at least one.
12. The device according to claim 9, in, The event problem fusion module includes: A prompt information acquisition unit, used to acquire standard prompt information, wherein the standard prompt information includes a slot of the event information and a question slot; An event information fusion unit, used to fill the event information into the slot of the event information; The question fusion unit is used to fill the question into the question slot to obtain a fusion result.
13. The device according to claim 9, further comprising: include: A real-time video acquisition module, used to acquire real-time video stream and acquisition information of the real-time video stream; An image acquisition module, used to acquire a plurality of collected images from the real-time video stream; The acquisition information acquisition module is used to determine the acquisition information of each acquired image according to the acquisition information of the real-time video stream; wherein the acquisition information includes the acquisition device identification, the acquisition device location and the acquisition time.
14. The device according to claim 13, in, The collection equipment includes traffic monitoring cameras within the city; the collection equipment location includes the shooting location of the traffic monitoring camera.
15. The device according to claim 9, in, The pre-trained large language model is determined by tuning a large language model for processing general tasks through instructions, and the large language model for processing general tasks is obtained through unsupervised training, labeled supervised fine-tuning, and human feedback reinforcement learning training.
16. The device according to claim 9, further comprising: include: A keyword extraction module, used to parse the question and obtain keywords; The reply storage module is used to store the reply content corresponding to the keyword and the question in correspondence, so as to obtain the reply content corresponding to the keyword for feedback when other questions corresponding to the keyword are received.
17. An electronic device, include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the event processing method based on a large language model according to any one of claims 1 to 8.
18. A non-transitory computer-readable storage medium storing computer instructions, in, The computer instructions are used to enable the computer to execute the event processing method based on a large language model according to any one of claims 1-8.
19. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the event processing method based on a large language model according to any one of claims 1 to 8.
Citation Information
Patent Citations
Video dense event description method based on multi-mode heterogeneous feature fusion
CN114359768A
Text event extraction method and device, equipment and storage medium
CN115510212A
Data extraction method and device based on sample model, equipment and storage medium
CN116151257A
Content interaction method and device, electronic equipment and storage medium
CN116821307A
Method for providing adaptive learning contents based on learning analytics
KR1020180016032A
