Data processing method and device, storage medium and electronic equipment

Abstract

The invention discloses a data processing method and device, a storage medium and electronic equipment. The method comprises the steps that a question text provided by a user is obtained, and the question text comprises a target keyword; a target vector is determined from a target vector library according to the problem text, the target vector represents a vector obtained by querying a target keyword, and each vector in the target vector library is obtained by executing vectorization operation through an analysis result output by the target multi-modal large model; the target multi-modal large model is used for determining an analysis result according to a target cue word, the target cue word is determined by a collection image and a preset cue word template, and the collection image represents a key frame image collected from an image collection device; and generating a target answer text based on the target vector, the target answer text being used for replying the question text. The technical problem that the data utilization rate is low due to the fact that the data extraction capacity is limited is solved.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to a data processing method and device, a storage medium, and an electronic device. Background Art

[0002] At present, in the relevant technology, the intelligent analysis of video images is mainly carried out through traditional structured algorithms. Since the existing algorithms have bottlenecks in extracting effective information from view data, for example, in a typical intersection scene, conventional structured algorithms can only extract structured information that is not related to humans and machines, while other objects, targets, scene information and most event information cannot be effectively utilized, resulting in a large amount of information resources being wasted. That is, there is a technical problem in the relevant technology that the utilization rate is low due to the limited data extraction capability.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present application provide a data processing method and device, a storage medium, and an electronic device to at least solve the technical problem of low data utilization due to limited data extraction capability.

[0005] According to one aspect of an embodiment of the present application, a data processing method is provided, comprising: obtaining a question text provided by a user, wherein the question text includes a target keyword; determining a target vector from a target vector library according to the question text, wherein the target vector represents a vector obtained by querying the target keyword, and each vector in the target vector library is obtained by performing a vectorization operation on a parsing result output by a target multimodal large model, and the target multimodal large model is used to determine the parsing result according to a target prompt word, and the target prompt word is determined by a captured image and a preset prompt word template, and the captured image represents a key frame image captured from an image acquisition device; generating a target answer text based on the target vector, wherein the target answer text is used to reply to the question text.

[0006] According to another aspect of an embodiment of the present application, a data processing device is also provided, including: an acquisition module, used to acquire a question text provided by a user, wherein the question text includes a target keyword; a determination module, used to determine a target vector from a target vector library according to the question text, wherein the target vector represents a vector obtained by querying the target keyword, and each vector in the target vector library is obtained by performing a vectorization operation on the parsing result output by a target multimodal large model, and the target multimodal large model is used to determine the parsing result according to a target prompt word, and the target prompt word is determined by a captured image and a preset prompt word template, and the captured image represents a key frame image captured from an image acquisition device; a generation module, used to generate a target answer text based on the target vector, wherein the target answer text is used to reply to the question text.

[0007] Optionally, the device is also used to: establish a connection relationship with the image acquisition device during the operation of the image acquisition service; in response to the connection relationship being established, determine the key frame image through a frame extraction operation during the process of the image acquisition device acquiring images; and input the key frame image into the target multimodal large model.

[0008] Optionally, the device is used to respond to the establishment of the connection relationship in the following manner: in the process of the image acquisition device acquiring images, the key frame image is determined by the frame extraction operation: in response to the establishment of the connection relationship, in the process of at least two of the image acquisition devices acquiring images, the first key frame image and the second key frame image are periodically determined by the frame extraction operation respectively; the device is used to input the key frame image into the target multimodal large model in the following manner: the first key frame image and the second key frame image are input into the target multimodal large model in sequence respectively to obtain a first analysis result and a second analysis result, wherein the first analysis result is stored in the target vector library according to the first image identifier, and the second analysis result is stored in the target vector library according to the second image identifier, the first image identifier includes the position information and time information corresponding to the first key frame image, and the second image identifier includes the position information and time information corresponding to the second key frame image.

[0009] Optionally, the device is also used to: generate the target prompt word according to the preset prompt word template and the key frame image, wherein the target prompt word includes address information corresponding to the key frame image, and the preset prompt word template is used to extract image content of the key frame image; input the target prompt word into the target multimodal large model to generate a target parsing result, wherein the target parsing result is used to represent the image content; and store the target parsing result in the target vector library.

[0010] Optionally, the device is used to store the target parsing result to the target vector library in the following manner: performing a preprocessing operation on the target parsing result to obtain a target text vector, wherein the target text vector represents a vectorized representation of the target parsing result; performing a standardization operation on the target text vector to obtain the target vector, and storing the target vector in the target vector library.

[0011] Optionally, the device is used to perform a preprocessing operation on the target parsing result to obtain a target text vector in the following manner: perform a word segmentation operation on the target text vector to obtain a group of words; perform a vector encoding operation on the group of words to obtain the target text vector.

[0012] Optionally, the device is used to determine the target vector from the target vector library according to the question text in the following manner: perform text extraction on the question text to determine the target keyword; send the target keyword to the target vector library, and determine the target vector in the target vector library based on the target keyword.

[0013] Optionally, the device is also used to: generate the target prompt word according to the preset prompt word template and the key frame image, wherein the target prompt word includes the address information corresponding to the key frame image, and the preset prompt word template is used to extract the target object of the key frame image; input the target prompt word into the target detection large model to generate a first target parsing result, wherein the first target parsing result is used to indicate the information associated with the target object in the key frame image, and the target multimodal large model includes the target detection large model; store the first target parsing result in the target vector library; the device is also used to: generate the target prompt word according to the preset prompt word template and the key frame image, wherein the target prompt word includes the address information corresponding to the key frame image, and the preset prompt word template is used to extract the target scene of the key frame image; input the target prompt word into the target detection large model to generate a first target parsing result, wherein the first target parsing result is used to indicate the information associated with the target object in the key frame image, and the target multimodal large model includes the target detection large model; store the first target parsing result in the target vector library. The target multimodal large model includes the scene detection large model, and generates a second target parsing result, wherein the second target parsing result is used to indicate the information associated with the target scene in the key frame image, and the target multimodal large model includes the scene detection large model; the second target parsing result is stored in the target vector library; the device is also used to: generate the target prompt word according to the preset prompt word template and the key frame image, wherein the target prompt word includes the address information corresponding to the key frame image, and the preset prompt word template is used to extract the target event in the key frame image; the target prompt word is input into the event detection large model to generate a third target parsing result, wherein the third target parsing result is used to indicate the information associated with the target event in the key frame image, and the target multimodal large model includes the event detection large model; the third target parsing result is stored in the target vector library.

[0014] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the above-mentioned data processing method when running.

[0015] According to another aspect of the embodiments of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above data processing method.

[0016] According to another aspect of the embodiments of the present application, there is also provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the data processing method through the computer program.

[0017] In an embodiment of the present application, a question text provided by a user is obtained, wherein the question text includes a target keyword; a target vector is determined from a target vector library based on the question text, wherein the target vector represents a vector obtained by querying the target keyword, and each vector in the target vector library is obtained by performing a vectorization operation on the parsing result output by the target multimodal large model, and the target multimodal large model is used to determine the parsing result based on the target prompt word, and the target prompt word is determined by a captured image and a preset prompt word template, and the captured image represents a key frame image captured from an image acquisition device; a target answer text is generated based on the target vector, wherein the target answer text is used to reply to the question text, and the purpose of accurately understanding and responding to user queries is achieved through deep parsing and vector mapping of the target multimodal large model, thereby achieving the technical effect of efficient extraction and utilization of video image information, and further solving the technical problem of low data utilization rate due to limited data extraction capability. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0019] Figure 1 is a schematic diagram of an application environment of an optional data processing method according to an embodiment of the present application;

[0020] Figure 2 is a flowchart of an optional data processing method according to an embodiment of the present application;

[0021] Figure 3 is a schematic diagram of an optional data processing method according to an embodiment of the present application;

[0022] Figure 4 is a schematic diagram of another optional data processing method according to an embodiment of the present application;

[0023] Figure 5 is a schematic diagram of another optional data processing method according to an embodiment of the present application;

[0024] Figure 6 is a schematic diagram of another optional data processing method according to an embodiment of the present application;

[0025] Figure 7 is a schematic diagram of another optional data processing method according to an embodiment of the present application;

[0026] Figure 8 is a schematic diagram of another optional data processing method according to an embodiment of the present application;

[0027] Fig. 9 is a schematic structural diagram of an optional data processing device according to an embodiment of the present application;

[0028] Fig.10 is a schematic diagram of the structure of an optional data processing product according to an embodiment of the present application;

[0029] Fig.11 It is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0031] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] The present application is described below in conjunction with embodiments:

[0033] According to one aspect of an embodiment of the present application, a data processing method is provided. Optionally, in this embodiment, the data processing method can be applied to Figure 1 In the hardware environment composed of the server 101 and the terminal device 103 shown in FIG. Figure 1As shown, the server 101 is connected to the terminal 103 via a network, and can be used to provide services for the terminal device or an application 107 installed on the terminal device. The application can be a video application, an instant messaging application, a browser application, an educational application, a game application, etc. A database 105 may be set up on the server or independently of the server to provide data storage services for the server 101, for example, a game data storage server. The above-mentioned network may include, but is not limited to, a wired network and a wireless network, wherein the wired network includes, for example, a local area network, a metropolitan area network, and a wide area network; the wireless network includes, for example, Bluetooth, WIFI, and other networks that implement wireless communication; the terminal device 103 may be a terminal configured with an application program, and may include, but is not limited to, at least one of the following: a mobile phone (such as an Android phone, an iOS phone, etc.), a laptop computer, a tablet computer, a PDA, a MID (Mobile Internet Devices), a PAD, a desktop computer, a smart TV, an intelligent voice interaction device, a smart home appliance, a vehicle-mounted terminal, an aircraft, a virtual reality (VR) terminal, an augmented reality (AR) terminal, a mixed reality (MR) terminal, and other computer devices; the above-mentioned server may be a single server, a server cluster consisting of multiple servers, or a cloud server.

[0034] Combination Figure 1 As shown, the above data processing method can be executed by an electronic device, which can be a terminal device or a server. The above data processing method can be implemented by the terminal device or the server respectively, or by the terminal device and the server together.

[0035] The above is only an example and is not specifically limited in this embodiment.

[0036] Optionally, as an optional implementation, as Figure 2 As shown, the method for processing the above data includes:

[0037] S202, obtaining a question text provided by a user, wherein the question text includes a target keyword;

[0038] Optionally, in an embodiment of the present application, the above-mentioned question text may include but is not limited to query requests, information retrieval, scene analysis, event query, target detection, etc. The above-mentioned target keywords may be set manually or randomly, and the present application does not impose any limitations.

[0039] Exemplarily, the user can provide the above-mentioned question text in the following ways, including but not limited to: direct input: the user can directly enter the question text in the search box or dialog box, such as entering a query request in an intelligent assistant or a search engine; voice recognition: the user can convert oral questions into text-based question text through a voice input device, such as a microphone; image recognition: the user can upload a picture containing text, and the system extracts the question text through optical character recognition technology; select preset questions: a series of common questions are preset in the system, and the user can provide question text by selecting preset questions; interactive dialogue: in a dialogue system, the user can gradually construct the question text through an interactive dialogue with the system; social media sharing: the user can share the question text on the social media platform, and the system obtains the text through crawling or user authorization; SMS or email sending: the user can provide the question text by sending SMS or email, and the system receives this information through an interface, etc.

[0040] In an exemplary embodiment, taking the intelligent video image analysis service system as an example, the user enters a question text in the search box: From 13:00 to 14:00 on day A, did car A pass through intersection B? First, the user's question text is parsed to extract the target keywords "from 13:00 to 14:00 on day A, car A, intersection B". Then, the extracted target keywords are sent to the vector library storage service. The vector library storage service retrieves the image vectors corresponding to 13:00 to 14:00 on day A and intersection B stored in history according to the target keywords, and determines whether car A exists at intersection B from 13:00 to 14:00 on day A.

[0041] Specifically, the target keywords can be identified from the question text in the following ways, including but not limited to: automatic extraction of the Qwen2.5-7B-Instruct chatglm3-6b language large model, keyword extraction algorithm, using algorithms such as TF-IDF (term frequency-inverse document frequency) to identify important words in the text; natural language processing (NLP) tools, using NLP libraries (such as NLTK, spaCy, Stanford NLP, etc.) for part-of-speech tagging, named entity recognition (NER), etc.

[0042] It should be noted that the above-mentioned target keyword can be composed of a single character or multiple characters, and a question text can include but is not limited to multiple target keywords, including but not limited to the time range, location range, event description, etc. of the user's question. The event description here can be multi-dimensional or single-dimensional, that is, the question text can be a question initiated about multiple events or a single event of an object, or a question initiated about multiple events or a single event of multiple objects. It can also be a question initiated about multiple events or a single event in an area or a time period, a question initiated about multiple events or a single event in an area or multiple time periods, or a question initiated about multiple events or a single event in multiple areas and a time period. This application does not limit this to ensure the flexibility of the question text. Users can learn about events that occurred in more than one object, one area, or one time period through a question text.

[0043] In another exemplary embodiment, the question text is "Did car A pass through intersection B from 13:00 to 14:00 on day A?" Then the time range is: 13:00 to 14:00 on day A; the location range is: intersection B; the event description is: car A; or, the question text is "Did a red truck pass through intersection B from 13:00 to 14:00 on day A?" Then the time range is: 13:00 to 14:00 on day A; the location range is: intersection B; the event description is: the vehicle type is a truck and the vehicle color is red; or, the question text is "Did a red truck and a woman wearing yellow clothes pass through intersection B from 13:00 to 14:00 on day A?" Then the time range is: 13:00 to 14:00 on day A; the location range is: intersection B; the event description is: the object type is a truck and a female pedestrian, the vehicle color is red, and the clothing color of the female pedestrian is yellow.

[0044] S204, determining a target vector from a target vector library according to the question text, wherein the target vector represents a vector obtained by querying a target keyword, and each vector in the target vector library is obtained by performing a vectorization operation on the parsing result output by the target multimodal large model, and the target multimodal large model is used to determine the parsing result according to the target prompt word, and the target prompt word is determined by a captured image and a preset prompt word template, and the captured image represents a key frame image captured from an image acquisition device;

[0045] Optionally, in an embodiment of the present application, the target vector library may be understood as a database storing multiple vectors, including but not limited to faiss, milvus, etc., which are obtained through the above-mentioned vectorization operation. The target vector refers to a vector obtained by querying the target keyword, which represents information related to the target keyword and is obtained from the parsing result of the target multimodal large model through the vectorization operation. The target multimodal large model may be understood as a neural network model that processes and understands multiple types of data (such as text, images, sounds, etc.) and generates parsing results based on the input data, including but not limited to qwen2-vl, MiniCPM-Llama3-V-2 _5Florence-2-large, etc. The above-mentioned vectorization operation refers to the process of converting data into vector form. The above-mentioned target prompt words refer to the data used to guide the target multimodal large model to perform parsing, including but not limited to text, voice, image, video, etc. The target prompt words help the model understand the data type and context that need to be processed. Specifically, it may include but is not limited to: setting roles (setting identities for the language large model, which helps the language large model provide more accurate and professional answers based on the target vector); providing background (providing background information that is helpful for answering questions); describing goals (accurately describing the goals you want to achieve); clarifying tasks (clarifying the tasks that need to be completed by the language large model), etc.

[0046] The above-mentioned preset prompt word template is a set of predefined data, including but not limited to text, voice, image, video, etc., which is used to guide the parsing process of the target multimodal large model. The preset prompt word template can be randomly set or manually pre-set to adapt to different application scenarios. The above-mentioned image acquisition equipment can include but is not limited to monitoring, cameras, camcorders, scanners, etc. The above-mentioned key frame image refers to a frame image selected from a continuous image sequence (video, etc.).

[0047] Exemplarily, the image acquisition device is composed of several cameras, and these cameras are all used to capture video. The key frame image can be an image obtained by periodically extracting frames from these cameras (for example, taking a screenshot of an image of a certain frame of the video, etc.), and the periods for periodically extracting the key frame images for cameras responsible for different services or deployed at different locations can be the same or different.

[0048] In an exemplary embodiment, assuming that the traffic volume at intersection A is greater than that at intersection B, the camera responsible for shooting intersection A can be set to extract a frame image from its own captured video every 2 seconds; the camera responsible for shooting intersection B can be set to extract a frame image from its own captured video every 1 second. That is, by adjusting the frame extraction period, computing and storage resources can be reasonably allocated according to actual business conditions to avoid excessive resource consumption at intersections with smaller traffic volumes.

[0049] In another exemplary embodiment, the period of extracting key frame images by the camera (the above-mentioned image acquisition device) in different time periods can be flexibly adjusted. For example, during the daytime period when there is heavy traffic, a more frequent extraction frequency is set to extract key frame images in the captured video; during the night period when there is less traffic, a more sparse extraction frequency is set to extract key frame images in the captured video, so as to achieve the technical effect of dynamically adjusting the monitoring strategy according to the actual traffic flow, improving the monitoring efficiency and resource utilization, while maintaining the real-time and flexibility of the monitoring.

[0050] Specifically, taking a camera in a target acquisition device to extract the above key frame image as an example, the implementation method of the camera periodically extracting the key frame image is as follows:

[0051] S1, define key frame image extraction rules: determine the frequency of key frame image extraction, for example, once every 10 seconds.

[0052] S2, determining the conditions for extracting key frame images, such as based on timestamp, scene change, motion detection, etc.

[0053] S3, initialize the camera: set the camera's resolution, frame rate and other acquisition parameters.

[0054] S4, key frame extraction: The camera captures a key frame image at each predetermined time point, and a timer can be set or a timestamp can be used to trigger the extraction of the key frame image.

[0055] In yet another exemplary embodiment, Figure 3 is a schematic diagram of an optional data processing method according to an embodiment of the present application, wherein the image acquisition service is as follows Figure 3 As shown:

[0056] S302, the server obtains the connection information of the video platform (connected to the image acquisition device, storing the acquired images) of the image data (captured images) from the database, including the IP address, port, user name, password, etc.;

[0057] S304, the server establishes a connection with the third-party system (video platform) according to the connection information of the video platform;

[0058] S306, the server obtains the connection result, if the connection fails, it goes to S308, if the connection is successful, it goes to S310;

[0059] S308, wait for 60 seconds and then return to S302 process;

[0060] S310, capturing key frame images of camera positions from a third-party system at regular intervals;

[0061] S312, push the key frame image to the large model analysis service for subsequent operations.

[0062] Exemplarily, the method may include but is not limited to performing text segmentation on the question text input by the user to obtain the above target keywords. Specifically, the question text is loaded through a loader, and the loaded question text is segmented into smaller units using a segmenter to obtain a number of text units, which may include but are not limited to words, characters, etc. The text unit may be a long participle or a short participle. The long participle helps to retain the semantic relationship of the context, and the short participle helps to retain the detailed information. The text unit may be segmented according to actual business needs. Then, the text embedding model is used to convert the several text units into numerical vectors, an index corresponding to the numerical vector is created, and a retriever is used to detect the target vector library based on the index to obtain the target vector. The specific implementation steps are as follows:

[0063] It should be noted that in the process of using a text embedding model to convert several text units into numerical vectors, some of the text vectors can be selected and converted into numerical vectors, and the original text units can be retained. For example, if the text unit represents the time information and position information of a key frame image, it does not need to be converted into a numerical vector. Instead, the remaining text units are converted into vectors. One text unit can correspond to one numerical vector, or the numerical vectors corresponding to the remaining text units can be spliced ​​to obtain an integrated numerical vector. This application does not limit this.

[0064] The numerical vector is compared with each vector in the target vector library to find the most relevant vector as the above-mentioned target vector. In the present application, the target vector is determined by vector query comparison rather than direct image comparison or text comparison, thereby reducing the comparison and calculation amount. Through vectorization, the semantic information of the text can be captured, rather than just the surface character matching, so as to achieve a more efficient, more accurate and more intelligent answer to the questions raised by the user. Alternatively, the question text entered by the user is input into the prompt word template to obtain the corresponding vector representation, and the vector representation is used to directly calculate the similarity with each image vector in the target vector library to obtain the corresponding target vector.

[0065] Specifically, by deploying an intelligent agent service, the intelligent agent service uses the langchain framework, including but not limited to using the langchain framework (a specific technical framework), receiving user natural language text query instructions (the above-mentioned question text), and analyzing the instructions to query its own tool set to decide which tool to use. If a knowledge base tool is required, the agent will extract the information in the user instruction (the above-mentioned target keyword) and call the knowledge base tool to perform a vector library storage service query (that is, querying the target vector in the target vector library according to the target keyword as mentioned above).

[0066] It should be noted that intelligent agent services refer to software programs that can perform specific tasks, process information, or interact with users. They can be chatbots, automation tools, or more complex artificial intelligence systems that can understand natural language, perform tasks, or provide decision support; the langchain framework is used to build and deploy intelligent agent services, which may include but are not limited to natural language processing, machine learning, dialogue management, task automation, user interface, etc.

[0067] In an exemplary embodiment, for example, the question text is: query a white car parked on the roadside with its door open from 2 to 4 p.m. yesterday. First, extract the target keywords from the question text, convert the extracted target keywords (the above-mentioned vectorization operation), and obtain a standardized result (vector representation) in a corresponding format. Then, query the target vector according to the standardized result. Figure 4 is a schematic diagram of an optional data processing method according to an embodiment of the present application. The workflow of the intelligent agent service is as follows Figure 4 As shown:

[0068] S402, parsing the natural language text query instruction input by the user to extract keyword information (target keyword);

[0069] S404, sending the keyword information to the target vector library storage service to obtain the target vector;

[0070] S406, answering the user's question based on the target vector library query result.

[0071] Exemplarily, the above-mentioned vectorization operations can be implemented, including but not limited to, in the following ways: Loop Unrolling: reducing the number of loop iterations and reducing loop control overhead by processing multiple elements at a time; Using a dedicated vector instruction set: modern processors support SIMD (Single Instruction Multiple Data) instruction sets, which can process multiple data at the same time; Library functions: Using libraries such as NumPy, Eigen, BLAS (Basic Linear Algebra Subprograms), which have optimized vector and matrix operations; Automatic vectorization: Some compilers (such as GCC and Clang) provide automatic vectorization options that can automatically convert certain loops into vectorized code during compilation.

[0072] Furthermore, the target vector library stores image vectors corresponding to several collected images. Taking an image in the collected images as an example, a prompt word template is first pre-set. The prompt word template is intended to extract several image information in the collected image. It is not limited to image information of a certain dimension, but image information of multiple dimensions in the collected image. For example, the prompt word template includes but is not limited to color features, shape features, text information, texture features, spatial relationships, object categories, actions or states, etc. For example, the prompt word template is "Does the picture include...", or "Please describe the content of the picture in detail", or "At () time, () character performed () action at () position, () character is () type, () color...", etc.; Specifically, the prompt word project can be understood as obtaining the best image interpretation effect through prompt word optimization, including: setting roles, setting identities for the target multimodal large model, which helps the large model provide more accurate and professional answers; providing background, providing background information that is helpful for answering questions; describing goals, accurately describing the goals to be achieved; clarifying tasks, clarifying the tasks that need to be completed by the large model, etc., and then, Figure 5 is a schematic diagram of an optional data processing method according to an embodiment of the present application. The processing process of the image vector corresponding to the image may include but is not limited to the following: Figure 5 As shown:

[0073] S502, start image acquisition service;

[0074] S504, inputting the image information of the collected image (which may be the collected image itself or the storage address of the collected image) into the prompt word template;

[0075] S506, generating a final prompt word (i.e., a model input of the target multimodal large model, the above-mentioned target prompt word) according to the prompt word template and the image information and inputting it into the target multimodal large model;

[0076] S508, the target multimodal large model outputs the parsing result according to the input prompt word and pushes it to the vector library storage service to perform subsequent operations. The parsing result here can be in the form of text, voice, image, video, etc.

[0077] It is understandable that the analysis results here include but are not limited to the description of the image content of the captured image, including the color, shape, object category, action or state, etc. in the image.

[0078] Furthermore, the vector library storage service may pre-process the parsing result to obtain an image vector corresponding to the captured image. Specifically, Figure 6 is a schematic diagram of an optional data processing method according to an embodiment of the present application. The processing flow of the vector library storage service can be as follows: Figure 6 As shown:

[0079] S602, start large model analysis service;

[0080] S604, receiving the parsing result of the target multimodal large model parsing service, and performing preprocessing on the parsing result, i.e., the above-mentioned vectorization operation (for example, using a text processing model to extract vectors from the parsing result, the text processing model includes but is not limited to a BERT model, etc., wherein there is a mapping relationship between the image vector and the collected image, which can be distinguished by an image identifier);

[0081] S606, storing the text embedding vector (the image vector corresponding to the captured image) generated after preprocessing into the vector database (target vector library).

[0082] S206, generating a target answer text based on the target vector, wherein the target answer text is used to reply to the question text.

[0083] Optionally, in an embodiment of the present application, the target answer text corresponds to the question text. For example, if the question text is “Did it rain yesterday?”, the target answer text is used to answer the question text, which may be “It did not rain yesterday and the weather was sunny.”

[0084] Exemplarily, the target vector may include multiple image vectors, each image vector corresponding to a respective captured image. For example, if the question text is “Did car A pass through intersection A between 14:00 and 16:00 yesterday afternoon?”, the target vector library may be queried based on the target keywords “14:00 to 16:00 yesterday afternoon, intersection A, car A”, the image vectors in the target vector library may be filtered based on the time range and location, and the event dimension (car A) information may be compared, and 50 image vectors matching the target keywords may be queried. The 50 captured images corresponding to these 50 image vectors are from videos captured by different cameras, and all captured intersection A between 14:00 and 16:00 yesterday afternoon. Car A exists in some of the captured images, and therefore, the corresponding target answer text may be “Car A passed through intersection A between 14:00 and 16:00 yesterday afternoon”.

[0085] In an exemplary embodiment, the data processing method proposed in the present application can be applied to a method and system for extracting holographic structured information from view data based on a multimodal large model. Figure 7 It is a method and system for extracting holographic structured information from view data based on a multimodal large model. This application can be roughly divided into two branches, such as Figure 7 The diagram includes a background image vector storage branch and a front-end user query branch. The background image vector storage branch may include but is not limited to:

[0086] S702, camera starts;

[0087] S704, start image acquisition service;

[0088] S706, capturing images from all cameras every minute and providing them to the large model analysis service;

[0089] S708, connecting to the large model analysis service;

[0090] S710, analyzing the image in real time, and storing the image description information (holographic structure), time, and point code into the target vector library;

[0091] S712, connect to the vector library storage service;

[0092] like Figure 7 As shown, the front-end user query branches may include but are not limited to:

[0093] S701, user logs in to the system query interface;

[0094] S703, the user uses natural language to query any person, thing, or object (for example, whether anyone held a banner on Qianmen Street yesterday);

[0095] S705, connecting to intelligent agent service;

[0096] S707, querying the description information (holographic structure) of all points in the target vector library according to the time range and location;

[0097] S709, returning the relevant description information (holographic structure), time, and point code stored in the target vector library to the intelligent agent service;

[0098] S711, the intelligent agent service generates a reply text based on the relevant description information and returns the query result (reply text) to the user.

[0099] Through the embodiments of the present application, a question text provided by a user is obtained, wherein the question text includes a target keyword; a target vector is determined from a target vector library according to the question text, wherein the target vector represents a vector obtained by querying the target keyword, and each vector in the target vector library is obtained by performing a vectorization operation on the parsing result output by the target multimodal large model, and the target multimodal large model is used to determine the parsing result according to the target prompt word, and the target prompt word is determined by a captured image and a preset prompt word template, and the captured image represents a key frame image captured from an image acquisition device; a target answer text is generated based on the target vector, wherein the target answer text is used to reply to the question text, and the purpose of accurately understanding and responding to user queries is achieved through deep parsing and vector mapping of the target multimodal large model, thereby achieving the technical effect of efficient extraction and utilization of video image information, and further solving the technical problem of low data utilization due to limited data extraction capabilities.

[0100] As an optional solution, the above method also includes: establishing a connection relationship with the above image acquisition device during the operation of the image acquisition service; in response to the above connection relationship being established, determining the above key frame image through a frame extraction operation during the process of the above image acquisition device acquiring images; and inputting the above key frame image into the above target multimodal large model.

[0101] Optionally, in an embodiment of the present application, the above-mentioned image acquisition service can communicate with an image acquisition device (such as a camera, scanner, etc.) after establishing a connection so as to receive acquired images transmitted from the image acquisition device. The image acquisition service can be understood as a software system or service responsible for managing and controlling the image acquisition device, and processing the image data received from the device. The image acquisition service may include user interface, data processing algorithms, storage management and other functions. The above-mentioned frame extraction operation refers to the selective extraction of specific frame images in a continuous image stream (video, etc.) according to certain conditions. The conditions here may include time, changes in image content, occurrence of specific events, etc.

[0102] For example, if the image acquisition service is a video surveillance system, the frame extraction operation may refer to extracting a frame at regular intervals in the video stream, or extracting key frames when motion is detected. These key frames can then be input into the target multimodal large model for further analysis and processing.

[0103] Through the embodiments of the present application, the frame extraction operation is utilized to achieve the purpose of reducing the amount of data, improving processing efficiency, and retaining key information.

[0104] As an optional scheme, in response to the establishment of the above-mentioned connection relationship, in the process of the above-mentioned image acquisition device acquiring images, the above-mentioned key frame image is determined by a frame extraction operation, including: in response to the establishment of the above-mentioned connection relationship, in the process of at least two of the above-mentioned image acquisition devices acquiring images, the first key frame image and the second key frame image are periodically determined by the above-mentioned frame extraction operation respectively; the above-mentioned key frame images are input into the above-mentioned target multimodal large model, including: the above-mentioned first key frame image and the above-mentioned second key frame image are input into the above-mentioned target multimodal large model in sequence, to obtain a first analysis result and a second analysis result, wherein the above-mentioned first analysis result is stored in the above-mentioned target vector library according to the above-mentioned first image identifier, and the above-mentioned second analysis result is stored in the above-mentioned target vector library according to the above-mentioned second image identifier, the above-mentioned first image identifier includes the position information and time information corresponding to the above-mentioned first key frame image, and the above-mentioned second image identifier includes the position information and time information corresponding to the above-mentioned second key frame image.

[0105] Optionally, in an embodiment of the present application, the first key frame image and the second key frame image are from videos captured by different image acquisition devices, and the periods of frame extraction operations corresponding to the first key frame image and the second key frame image may be the same or different.

[0106] Optionally, the first image identifier and the second image identifier are used to distinguish different key frame images, the time information of which refers to the shooting time of the key frame image, and the location information refers to the location area where the key frame image is shot. The first analysis result includes but is not limited to the image content description of the first key frame image, including the color, shape, object category, action or state in the image; the second analysis result includes but is not limited to the image content description of the second key frame image, including the color, shape, object category, action or state in the image.

[0107] Exemplarily, the first key frame image and the second key frame image can be respectively input into the target multimodal large model to obtain their respective corresponding analysis results, i.e., the first analysis result and the second analysis result. The first analysis result is stored in the target vector library according to the image identification of the first key frame image. Similarly, the second analysis result is stored in the target vector library according to the image identification of the second key frame image, so that the analysis result of each image can be retrieved and used according to its identification.

[0108] Exemplarily, the information stored in the target vector library may include but is not limited to the following table:

[0109]

[0110] Furthermore, each vector in the target vector library is marked according to the image identifier. Taking a key frame image as an example, the image identifier of the key frame image includes time information (indicating that the key frame image was taken at 10:00:00 on XX day) and location information (indicating that the key frame image was taken in area A of intersection), and the image content in the table is represented by a vector, which is the vector in the target vector library. When searching for the target vector, the time information and location information in the target keyword are first obtained, and the image identifier is determined based on the two. By comparing the vectors in the target vector library, the vector that meets the time information requirements and the location information requirements is first determined. For example, if the vector in the target vector library corresponds to the key frame image taken at 10:00:00 on XX day, it is considered to meet the time information requirements; if the vector in the target vector library corresponds to the key frame image taken in area A of intersection, it is considered to meet the location information requirements.

[0111] After the above operations, it is assumed that N vectors that meet both the time information requirements and the location information requirements are screened out from the target vector library. Further, the numerical vector corresponding to the above keywords is compared with these N vectors to find the most relevant vector as the above target vector, thereby effectively reducing the number of vectors that need to be compared for similarity, speeding up the vector comparison efficiency, and thus improving the retrieval efficiency of the target vector.

[0112] As an optional solution, the method further includes: generating the target prompt word according to the preset prompt word template and the key frame image, wherein the target prompt word includes the address information corresponding to the key frame image, and the preset prompt word template is used to extract the image content of the key frame image; inputting the target prompt word into the target multimodal large model to generate a target parsing result, wherein the target parsing result is used to represent the image content; and storing the target parsing result into the target vector library.

[0113] Optionally, in an embodiment of the present application, the above-mentioned address information refers to the storage location of the above-mentioned key frame image, the above-mentioned time information refers to the frame time of the above-mentioned key frame image, and the above-mentioned image content may include image information of multiple dimensions, for example, the object type, object position, object status, etc. in the captured image.

[0114] In an exemplary embodiment, the preset prompt word template includes "identify objects in the image" and "extract metadata from the image". Assuming that the image acquisition device is a plurality of surveillance cameras, the key frame image is a video frame image in the video acquired by the surveillance camera. For example, the information extracted from the key frame image may be "the 10th frame image 1.jpg is located on server A and shows a red car"; the target multimodal large model receives the address information of "image 1.jpg", analyzes the image content, identifies the red car, and the corresponding target parsing result output by the model is "image 1.jpg includes a red car".

[0115] As an optional solution, the above-mentioned storing the above-mentioned target parsing result into the above-mentioned target vector library includes: performing a preprocessing operation on the above-mentioned target parsing result to obtain a target text vector, wherein the above-mentioned target text vector represents a vectorized representation of the above-mentioned target parsing result; performing a standardization operation on the above-mentioned target text vector to obtain the above-mentioned target vector, and storing the above-mentioned target vector into the above-mentioned target vector library.

[0116] Optionally, in an embodiment of the present application, the above-mentioned target text vector may include but is not limited to a vector obtained by using a Word2Vec (text vector conversion) model, or a vector obtained by other natural language processing (NLP) technologies.

[0117] In an exemplary embodiment, the target text vector may be normalized by using the L2 norm so that the length of the vector is 1, and then the normalized target vector is stored in a target vector library, wherein the L2 norm (also called the Euclidean norm or Euclidean distance) is a method of measuring the length of a vector and is used to represent the size of a vector in the Euclidean space.

[0118] Through the embodiments of the present application, effective representation and processing of target parsing results are achieved, providing support for subsequent querying of target vectors in a target vector library.

[0119] As an optional solution, the above-mentioned preprocessing operation is performed on the above-mentioned target parsing result to obtain the target text vector, including: performing a word segmentation operation on the above-mentioned target text vector to obtain a group of words; performing a vector encoding operation on the above-mentioned group of words to obtain the above-mentioned target text vector.

[0120] Optionally, in an embodiment of the present application, the above-mentioned word segmentation operation is a basic step in natural language processing (NLP), which is used to decompose continuous text strings into smaller units, such as words, phrases or symbols, and can be implemented including but not limited to the following methods: rule-based methods, regular expressions: use regular expressions to match and segment words or symbols in text; dictionary matching: use a predefined dictionary to identify and segment words in text. Statistical-based methods, N-gram model: segment text by counting the frequency of occurrence of N consecutive characters or words in the text; Hidden Markov Model (HMM): use state transition probabilities to determine the boundaries of words in text. Machine learning-based methods, support vector machine (SVM): train a classifier to identify word boundaries in text; conditional random field (CRF): a sequence model used to learn the probability distribution of word boundaries in text, etc.

[0121] Optionally, in an embodiment of the present application, the vector encoding operation is also a basic step in natural language processing (NLP), which is used to convert text data into a numerical form, and can be implemented including but not limited to the following methods: Bag of Words (BoW): converting text into a vector of the number of word occurrences; TF-IDF (Term Frequency-Inverse Document Frequency): considering the frequency of words in documents and their distribution in the entire corpus; Word Embeddings: such as Word2Vec, GloVe, etc., mapping words to vectors in a high-dimensional space; Sentence encoding: such as BERT, GPT, etc., converting an entire sentence or paragraph into a vector of a fixed length, etc.

[0122] For example, assuming that the target parsing result is "A car was parked at the door at 2 p.m. yesterday", first, the text is segmented into words to obtain: yesterday at 2 p.m., the door, a car was parked, then each word is vector encoded to convert each word into a vector form, and finally, the vectors corresponding to each word are combined to form a vector representation of the entire text, that is, the above-mentioned target text vector, which can be achieved including but not limited to splicing, averaging or using more complex models such as BERT.

[0123] As an optional solution, the above-mentioned determining the target vector from the target vector library based on the above-mentioned question text includes: performing text extraction on the above-mentioned question text to determine the above-mentioned target keywords; sending the above-mentioned target keywords to the above-mentioned target vector library, and determining the above-mentioned target vector based on the above-mentioned target keywords in the above-mentioned target vector library.

[0124] Optionally, in an embodiment of the present application, the above-mentioned text extraction refers to identifying and extracting useful information or keywords, ie, the above-mentioned target keywords, from the text.

[0125] Exemplarily, the question text is analyzed to identify target keywords; the target keywords are sent to a target vector library; in the target vector library, the corresponding target vector is retrieved or calculated according to the target keywords.

[0126] As an optional solution, the method further includes at least one of the following: generating the target prompt word according to the preset prompt word template and the key frame image, wherein the target prompt word includes the address information corresponding to the key frame image, and the preset prompt word template is used to extract the target object of the key frame image; inputting the target prompt word into the target detection large model to generate a first target parsing result, wherein the first target parsing result is used to indicate the information associated with the target object in the key frame image, and the target multimodal large model includes the target detection large model; storing the first target parsing result into the target vector library; generating the target prompt word according to the preset prompt word template and the key frame image, wherein the target prompt word includes the address information corresponding to the key frame image, and the preset prompt word template is used to extract the target scene of the key frame image; The target prompt word is input into the scene detection big model to generate a second target parsing result, wherein the second target parsing result is used to indicate the information associated with the target scene in the key frame image, and the target multimodal big model includes the scene detection big model; the second target parsing result is stored in the target vector library; the target prompt word is generated according to the preset prompt word template and the key frame image, wherein the target prompt word includes the address information corresponding to the key frame image, and the preset prompt word template is used to extract the target event in the key frame image; the target prompt word is input into the event detection big model to generate a third target parsing result, wherein the third target parsing result is used to indicate the information associated with the target event in the key frame image, and the target multimodal big model includes the event detection big model; the third target parsing result is stored in the target vector library.

[0127] Optionally, in an embodiment of the present application, the target object may include but is not limited to objects appearing in the key frame image, for example, people, vehicles, buildings, plants, pets, etc.; the first target parsing result may be represented in a vectorized form, and the first target parsing result may include but is not limited to information such as the object type, object appearance, object state, and object size of the target object; the object detection model may include but is not limited to a convolutional neural network model, a recurrent neural network model, etc.

[0128] For example, Figure 8 is a schematic diagram of another optional data processing method according to an embodiment of the present application, such as Figure 8 As shown:

[0129] S802, start image acquisition service;

[0130] S804, inputting the image information of the collected image (which may be the collected image itself or the storage address of the collected image) into the prompt word template;

[0131] S806-1, generating a target prompt word according to the prompt word template and the image information, as an input of the object detection model branch in the target multimodal large model;

[0132] S808, the object detection model branch outputs a first parsing result according to the input prompt word and pushes it to the vector library storage service to perform subsequent operations. The parsing result here can be in the form of vector, text, voice, image, video, etc.

[0133] Optionally, in an embodiment of the present application, the above-mentioned target scene may include but is not limited to the scene appearing in the above-mentioned key frame image, for example, a school scene, a hospital scene, a traffic scene, etc.; the above-mentioned second target parsing result may be represented in a vectorized form, and the second target parsing result may include but is not limited to the scene type, scene scale, characters and objects in the scene and other information of the target scene; the above-mentioned scene detection large model may include but is not limited to a convolutional neural network model, a recurrent neural network model, etc.

[0134] For example, Figure 8 As shown:

[0135] S802, start image acquisition service;

[0136] S804, inputting the image information of the collected image (which may be the collected image itself or the storage address of the collected image) into the prompt word template;

[0137] S806-2, generating a target prompt word according to the prompt word template and the image information, as an input of the scene detection model branch in the target multimodal large model;

[0138] S808, the scene detection model branch outputs a second parsing result based on the input prompt word and pushes it to the vector library storage service to perform subsequent operations. The parsing result here can be in the form of vector, text, voice, image, video, etc.

[0139] Optionally, in an embodiment of the present application, the above-mentioned target events may include but are not limited to events appearing in the above-mentioned key frame images, for example, running events, singing events, sudden traffic events, etc.; the above-mentioned third target analysis results can be represented in a vectorized form, and the third target analysis results may include but are not limited to the event type, event scale, people and objects in the event and other information of the target event; the above-mentioned event detection large model may include but is not limited to a convolutional neural network model, a recurrent neural network model, etc.

[0140] For example, Figure 8 is a schematic diagram of another optional data processing method according to an embodiment of the present application, such as Figure 8 As shown:

[0141] S802, start image acquisition service;

[0142] S804, inputting the image information of the collected image (which may be the collected image itself or the storage address of the collected image) into the prompt word template;

[0143] S806-3, generating a target prompt word according to the prompt word template and the image information, as an input of the event detection model branch in the target multimodal large model;

[0144] S808, the event detection model branch outputs a third parsing result according to the input prompt word and pushes it to the vector library storage service to perform subsequent operations. The parsing result here can be in the form of vector, text, voice, image, video, etc.

[0145] Furthermore, the object detection model branch, the scene detection model branch, and the event detection model branch may be executed simultaneously, and the vector finally generated may include but is not limited to representing objects, scenes, events, etc. in the key frame image.

[0146] In an exemplary embodiment, the data processing scheme proposed in this application can be applied to the field of video AI analysis technology, for example, in a method and system for extracting holographic structured information from view data based on a multimodal large model, considering that the existing video intelligent system in the related art intelligently extracts a small part of the information of the image, and can only use some preset attributes when searching. As a result, a large amount of available information in the video is not extracted, and the user cannot freely search for the target and event they want to find. Therefore, this application provides an image acquisition service, customizes the key frame data of the video point from the video system, inputs the data into the large model parsing service, outputs the corresponding structured information data, and then stores the data in the vector library storage service. The user uses natural language to send a query instruction to the intelligent agent service, and the intelligent agent service obtains the query result from the vector library service and returns it to the user; on the one hand, it greatly improves the amount of structured information extracted from the video image, and on the other hand, it also improves the ease of use of the entire system, so that users can use natural language to interact with the entire system, and can meet the city-level people and vehicles in the whole city range and all time periods. Rapid retrieval.

[0147] Specifically, the embodiments of the present application may include but are not limited to image acquisition services, large model analysis services, vector library storage services, and intelligent agent services. The services ultimately complete the extraction, storage, and natural language query of image holographic structured information through interlocking business logic. Among them, the image acquisition service connects to the video platform through the configured video platform authentication information, regularly captures key frame images of video points on the video surveillance platform, and pushes them to the large model analysis service.

[0148] S1. The core of the large model analysis service is the pre-trained multimodal large model. The multimodal large model is a machine learning technology based on deep learning. Its core idea is to fuse different media data (such as text, images, audio and video, etc.) and learn the relationship between different modalities to achieve more intelligent information processing. In the multimodal large model, data of different modalities are input into a deep neural network after preprocessing, and finally output the corresponding results after multiple layers of feature extraction and fusion. The large model analysis service first generates the final prompt word based on the pre-made prompt word template, and then uses the multimodal large model to extract all the descriptive information of targets, events, etc. in the picture according to the prompt word instructions, and sends the information to the vector library storage service for storage;

[0149] S2, after the vector library storage service receives the image holographic structured information, it performs text segmentation on the information. Text segmentation mainly considers two factors: 1) the token limit of the embedding model; 2) the impact of semantic integrity on the overall retrieval effect. Some common text segmentation methods are as follows: Sentence segmentation: segment at the granularity of sentences to retain the complete semantics of a sentence. Common segmentation symbols include: period, exclamation mark, question mark, line break, etc. Fixed-length segmentation: According to the token length limit of the embedding model, the text is segmented into fixed lengths (for example, 256 / 512 tokens). This segmentation method will lose a lot of semantic information, which is generally alleviated by adding a certain amount of redundancy at the head and tail. After text segmentation, each block of text must be vectorized. Vectorization is a process of converting text data into a vector matrix, which will directly affect the subsequent retrieval effect. Currently, an open source embedding model is used for text vectorization conversion;

[0150] S3, the intelligent agent service uses the large model agent as the core module. The large model agent is an intelligent body built on a large language model (LLM). It has the ability to perceive the environment, autonomously understand, make decisions and act. The agent is able to simulate independent thinking processes, flexibly call various tools, and gradually achieve preset goals. In terms of technical architecture, the agent changes from a process-oriented architecture to a goal-oriented architecture, aiming to complete complex tasks through the close combination of perception, thinking and action. The intelligent agent service uses the langchain framework. First, it receives the user's natural language text query instructions, analyzes the instructions and queries its own tool set to decide which tool to use. If a knowledge base tool is needed, the agent will extract the information in the user's instructions and call the knowledge base tool to query the vector library storage service. After receiving the query result, the agent will check the result to determine whether the result is correct. If it is not correct, it will continue to call the tool query. If it is correct, it will format the query result and return it to the user.

[0151] For example, in the embodiments of the present application, it may include but is not limited to:

[0152] S1, image acquisition service:

[0153] S101, obtaining connection information of the video platform of the image data from the database, including IP address, port, user name, password, etc.;

[0154] S102, establishing a connection with the video platform according to the connection information of the video platform;

[0155] S103, obtain the connection result. If the connection fails, wait for 60 seconds and return to S102; if the connection is successful, enter S104;

[0156] S104, capturing key frame images of camera positions from the video platform at regular intervals;

[0157] S105, pushing the key frame image to the large model parsing service;

[0158] S2, large model analysis service:

[0159] S201, inputting the picture information input by the image acquisition service into the prompt word template;

[0160] S202, generating a final prompt word according to the prompt word template and the image information and inputting it into the multimodal large model;

[0161] S203, the multimodal large model outputs the parsing result according to the input prompt word and pushes it to the target vector library storage service;

[0162] S3, vector storage service:

[0163] S301, receiving the analysis result of the large model analysis service and preprocessing the result;

[0164] S302, storing the text embedding vector generated after preprocessing into a target vector library;

[0165] S4, intelligent agent service:

[0166] S401, parsing the natural language text query instruction input by the user to extract keyword information;

[0167] S402, sending keyword information to the target vector library storage service to obtain relevant results;

[0168] S403, answering the user's question according to the target vector library query result;

[0169] Through the embodiments of the present application, the multimodal large model technology is used to parse the view information, thereby increasing the amount of structured information in the view information;

[0170] By leveraging the large model's ability to understand natural language, users can interact with the system using natural language, greatly improving the convenience of system interaction; and, using vector database technology, relevant semantic data can be queried.

[0171] It is understandable that in the specific implementation of this application, related data such as user information is involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0172] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0173] According to another aspect of the embodiment of the present application, a data processing device for implementing the above-mentioned data processing method is also provided. Fig. 9 As shown, the device comprises:

[0174] An acquisition module 902 is used to acquire a question text provided by a user, wherein the question text includes a target keyword;

[0175] A determination module 904 is used to determine a target vector from a target vector library according to the question text, wherein the target vector represents a vector obtained by querying a target keyword, and each vector in the target vector library is obtained by performing a vectorization operation on the parsing result output by the target multimodal large model, and the target multimodal large model is used to determine the parsing result according to the target prompt word, and the target prompt word is determined by a captured image and a preset prompt word template, and the captured image represents a key frame image captured from an image acquisition device;

[0176] The generating module 906 is used to generate a target answer text based on the target vector, wherein the target answer text is used to reply to the question text.

[0177] As an optional solution, the above-mentioned device is also used to: establish a connection relationship with the above-mentioned image acquisition device during the operation of the image acquisition service; in response to the above-mentioned connection relationship being established, determine the above-mentioned key frame image through frame extraction operation during the process of the above-mentioned image acquisition device acquiring images; and input the above-mentioned key frame image into the above-mentioned target multimodal large model.

[0178] As an optional scheme, the above-mentioned device is used to respond to the establishment of the above-mentioned connection relationship in the following manner, and determine the above-mentioned key frame image by frame extraction operation during the process of the above-mentioned image acquisition device acquiring images: in response to the establishment of the above-mentioned connection relationship, in the process of at least two of the above-mentioned image acquisition devices acquiring images, the first key frame image and the second key frame image are periodically determined by the above-mentioned frame extraction operation respectively; the above-mentioned device is used to input the above-mentioned key frame images into the above-mentioned target multimodal large model in the following manner: the above-mentioned first key frame image and the above-mentioned second key frame image are input into the above-mentioned target multimodal large model in sequence to obtain a first analysis result and a second analysis result, wherein the above-mentioned first analysis result is stored in the above-mentioned target vector library according to the above-mentioned first image identifier, and the above-mentioned second analysis result is stored in the above-mentioned target vector library according to the above-mentioned second image identifier, the above-mentioned first image identifier includes the position information and time information corresponding to the above-mentioned first key frame image, and the above-mentioned second image identifier includes the position information and time information corresponding to the above-mentioned second key frame image.

[0179] As an optional solution, the above-mentioned device is also used to: generate the above-mentioned target prompt word according to the above-mentioned preset prompt word template and the above-mentioned key frame image, wherein the above-mentioned target prompt word includes the address information corresponding to the above-mentioned key frame image, and the above-mentioned preset prompt word template is used to extract the image content of the above-mentioned key frame image; input the above-mentioned target prompt word into the above-mentioned target multimodal large model to generate a target parsing result, wherein the above-mentioned target parsing result is used to represent the above-mentioned image content; and store the above-mentioned target parsing result into the above-mentioned target vector library.

[0180] As an optional solution, the above-mentioned device is used to store the above-mentioned target parsing result into the above-mentioned target vector library in the following manner: performing a preprocessing operation on the above-mentioned target parsing result to obtain a target text vector, wherein the above-mentioned target text vector represents a vectorized representation of the above-mentioned target parsing result; performing a standardization operation on the above-mentioned target text vector to obtain the above-mentioned target vector, and storing the above-mentioned target vector into the above-mentioned target vector library.

[0181] As an optional solution, the above-mentioned device is used to perform preprocessing operations on the above-mentioned target parsing results in the following manner to obtain the target text vector: perform word segmentation operations on the above-mentioned target text vector to obtain a group of words; perform vector encoding operations on the above-mentioned group of words to obtain the above-mentioned target text vector.

[0182] As an optional solution, the above-mentioned device is used to determine the target vector from the target vector library based on the above-mentioned question text in the following manner: perform text extraction on the above-mentioned question text to determine the above-mentioned target keywords; send the above-mentioned target keywords to the above-mentioned target vector library, and determine the above-mentioned target vector based on the above-mentioned target keywords in the above-mentioned target vector library.

[0183] As an optional solution, the above-mentioned device is also used to: generate the above-mentioned target prompt word according to the above-mentioned preset prompt word template and the above-mentioned key frame image, wherein the above-mentioned target prompt word includes the address information corresponding to the above-mentioned key frame image, and the above-mentioned preset prompt word template is used to extract the target object of the above-mentioned key frame image; input the above-mentioned target prompt word into the target detection large model to generate a first target parsing result, wherein the above-mentioned first target parsing result is used to indicate the information associated with the above-mentioned target object in the above-mentioned key frame image, and the above-mentioned target multimodal large model includes the above-mentioned target detection large model; store the above-mentioned first target parsing result in the above-mentioned target vector library; the above-mentioned device is also used to: generate the above-mentioned target prompt word according to the above-mentioned preset prompt word template and the above-mentioned key frame image, wherein the above-mentioned target prompt word includes the address information corresponding to the above-mentioned key frame image, and the above-mentioned preset prompt word template is used to extract the target scene of the above-mentioned key frame image; input the above-mentioned target prompt word into the target detection large model to generate a first target parsing result, wherein the above-mentioned first target parsing result is used to indicate the information associated with the above-mentioned target object in the above-mentioned key frame image, and the above-mentioned target multimodal large model includes the above-mentioned target detection large model; store the above-mentioned first target parsing result in the above-mentioned target vector library. The target word is input into the scene detection big model to generate a second target analysis result, wherein the second target analysis result is used to indicate the information associated with the target scene in the key frame image, and the target multimodal big model includes the scene detection big model; the second target analysis result is stored in the target vector library; the device is also used to: generate the target prompt word according to the preset prompt word template and the key frame image, wherein the target prompt word includes the address information corresponding to the key frame image, and the preset prompt word template is used to extract the target event in the key frame image; the target prompt word is input into the event detection big model to generate a third target analysis result, wherein the third target analysis result is used to indicate the information associated with the target event in the key frame image, and the target multimodal big model includes the event detection big model; the third target analysis result is stored in the target vector library.

[0184] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0185] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0186] According to one aspect of the present application, a computer program product is provided. The computer program product includes a computer program.

[0187] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0188] Fig.10 The structure block diagram of a computer system for implementing an electronic device according to an embodiment of the present application is schematically shown.

[0189] It should be noted that Fig.10 The computer system 1000 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0190] like Fig.10 As shown, the computer system 1000 includes a central processing unit 1001 (CPU), which can perform various appropriate actions and processes according to the program stored in the read-only memory 1002 (ROM) or the program loaded from the storage part 1008 to the random access memory 1003 (RAM). Various programs and data required for system operation are also stored in the random access memory 1003. The central processing unit 1001, the read-only memory 1002 and the random access memory 1003 are connected to each other through a bus 1004. The input / output interface 1005 (Input / Output interface, i.e., I / O interface) is also connected to the bus 1004.

[0191] The following components are connected to the input / output interface 1005: an input section 1006 including a keyboard, a mouse, etc.; an output section 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, a modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the input / output interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed, so that a computer program read therefrom is installed into the storage section 1008 as needed.

[0192] In particular, according to an embodiment of the present application, the process described in each method flow chart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer readable medium, and the computer program contains a program code for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1009, and / or installed from the removable medium 1011. When the computer program is executed by the central processor 1001, various functions defined in the system of the present application are executed.

[0193] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1009, and / or installed from the removable medium 1011. When the computer program is executed by the central processor 1001, various functions provided by the embodiments of the present application are executed.

[0194] According to another aspect of the embodiment of the present application, an electronic device for implementing the above data processing method is also provided. The electronic device may be Figure 1 The terminal device or server shown in the figure. This embodiment is described by taking the electronic device as a terminal device as an example. Fig.11 As shown, the electronic device includes a memory 1102 and a processor 1104. The memory 1102 stores a computer program, and the processor 1104 is configured to execute the steps in any of the above method embodiments through the computer program.

[0195] Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices of a computer network.

[0196] Optionally, in this embodiment, the above-mentioned processor can be configured to execute the methods in each embodiment of the present application through a computer program.

[0197] Alternatively, a person skilled in the art may understand that: Fig.11 The structure shown is for illustration only. Fig.11 The structure of the electronic device is not limited. Fig.11 More or fewer components (such as network interfaces, etc.) as shown in, or with Fig.11 Different configurations are shown.

[0198] Among them, the memory 1102 can be used to store software programs and modules, such as the program instructions / modules corresponding to the data processing method and device in the embodiments of the present application. The processor 1104 executes various functional applications and data processing by running the software programs and modules stored in the memory 1102, that is, realizing the above-mentioned data processing method. The memory 1102 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 1102 may further include a memory remotely located relative to the processor 1104, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. Among them, the memory 1102 can be specifically, but not limited to, used to store information such as key frame images. As an example, such as Fig.11 As shown, the memory 1102 may include but is not limited to the acquisition module 902, determination module 904, and generation module 906 in the data processing device. In addition, it may also include but is not limited to other module units in the data processing device, which will not be repeated in this example.

[0199] Optionally, the transmission device 1106 is used to receive or send data via a network. Specific examples of the network may include a wired network and a wireless network. In one example, the transmission device 1106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices and routers via a network cable so as to communicate with the Internet or a local area network. In one example, the transmission device 1106 is a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0200] In addition, the electronic device further includes: a display 1108 for displaying the key frame image; and a connection bus 1111 for connecting various module components in the electronic device.

[0201] In other embodiments, the terminal device or server may be a node in a distributed system, wherein the distributed system may be a blockchain system, and the blockchain system may be a distributed system formed by connecting the multiple nodes through network communication. The nodes may form a peer-to-peer network, and any form of computing device, such as a server, terminal or other electronic device, may become a node in the blockchain system by joining the peer-to-peer network.

[0202] According to one aspect of the present application, a computer-readable storage medium is provided, and a processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device executes the data processing method provided in various optional implementations of the above-mentioned data processing aspects.

[0203] Optionally, in this embodiment, the above-mentioned computer-readable storage medium can be configured to store data for executing the methods in various embodiments of the present application.

[0204] Optionally, in this embodiment, a person of ordinary skill in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing hardware related to the terminal device through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0205] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0206] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for one or more electronic devices to execute all or part of the steps of the methods described in each embodiment of the present application.

[0207] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0208] In the several embodiments provided in the present application, it should be understood that the disclosed application can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0209] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0210] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0211] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A data processing method, characterized in that: include: Acquire a question text provided by a user, wherein the question text includes a target keyword; Determine a target vector from a target vector library according to the question text, wherein the target vector represents a vector obtained by querying the target keyword, each vector in the target vector library is obtained by performing a vectorization operation on the parsing result output by the target multimodal large model, the target multimodal large model is used to determine the parsing result according to a target prompt word, the target prompt word is determined by a captured image and a preset prompt word template, and the captured image represents a key frame image captured from an image acquisition device; A target answer text is generated based on the target vector, wherein the target answer text is used to reply to the question text.

2. The method according to claim 1, characterized in that The method further comprises: During the operation of the image acquisition service, establishing a connection relationship with the image acquisition device; In response to the connection relationship being established, during the process of the image acquisition device acquiring images, determining the key frame image by a frame extraction operation; The key frame image is input into the target multimodal large model.

3. The method according to claim 2, characterized in that In response to the connection relationship being established, in the process of the image acquisition device acquiring images, determining the key frame image by a frame extraction operation, including: in response to the connection relationship being established, in the process of at least two of the image acquisition devices acquiring images, periodically determining a first key frame image and a second key frame image by the frame extraction operation respectively; Inputting the key frame image into the target multimodal large model includes: inputting the first key frame image and the second key frame image into the target multimodal large model in sequence, respectively, to obtain a first analysis result and a second analysis result, wherein the first analysis result is stored in the target vector library according to a first image identifier, and the second analysis result is stored in the target vector library according to a second image identifier, the first image identifier includes position information and time information corresponding to the first key frame image, and the second image identifier includes position information and time information corresponding to the second key frame image.

4. The method according to claim 1, characterized in that The method further comprises: generating the target prompt word according to the preset prompt word template and the key frame image, wherein the target prompt word includes address information corresponding to the key frame image, and the preset prompt word template is used to extract image content of the key frame image; Inputting the target prompt word into the target multimodal macro model to generate a target parsing result, wherein the target parsing result is used to represent the image content; The target parsing result is stored in the target vector library.

5. The method according to claim 4, characterized in that The storing the target parsing result into the target vector library comprises: Performing a preprocessing operation on the target parsing result to obtain a target text vector, wherein the target text vector represents a vectorized representation of the target parsing result; A standardization operation is performed on the target text vector to obtain the target vector, and the target vector is stored in the target vector library.

6. The method according to claim 5, characterized in that The performing of a preprocessing operation on the target parsing result to obtain a target text vector includes: Performing a word segmentation operation on the target text vector to obtain a group of words; A vector encoding operation is performed on the group of words to obtain the target text vector.

7. The method according to claim 1, characterized in that The step of determining a target vector from a target vector library according to the question text comprises: Performing text extraction on the question text to determine the target keyword; The target keyword is sent to the target vector library, and the target vector is determined in the target vector library based on the target keyword.

8. The method according to claim 1, characterized in that The method further comprises at least one of the following: Generate the target prompt word according to the preset prompt word template and the key frame image, wherein the target prompt word includes address information corresponding to the key frame image, and the preset prompt word template is used to extract the target object of the key frame image; input the target prompt word into the target detection large model to generate a first target parsing result, wherein the first target parsing result is used to indicate information associated with the target object in the key frame image, and the target multimodal large model includes the target detection large model; store the first target parsing result in the target vector library; Generate the target prompt word according to the preset prompt word template and the key frame image, wherein the target prompt word includes address information corresponding to the key frame image, and the preset prompt word template is used to extract the target scene of the key frame image; input the target prompt word into the scene detection large model to generate a second target parsing result, wherein the second target parsing result is used to indicate information associated with the target scene in the key frame image, and the target multimodal large model includes the scene detection large model; store the second target parsing result in the target vector library; The target prompt word is generated according to the preset prompt word template and the key frame image, wherein the target prompt word includes address information corresponding to the key frame image, and the preset prompt word template is used to extract the target event in the key frame image; the target prompt word is input into the event detection large model to generate a third target parsing result, wherein the third target parsing result is used to indicate information associated with the target event in the key frame image, and the target multimodal large model includes the event detection large model; the third target parsing result is stored in the target vector library.

9. A data processing device, characterized in that: include: An acquisition module, used to acquire a question text provided by a user, wherein the question text includes a target keyword; a determination module, configured to determine a target vector from a target vector library according to the question text, wherein the target vector represents a vector obtained by querying the target keyword, each vector in the target vector library is obtained by performing a vectorization operation on the parsing result output by the target multimodal large model, the target multimodal large model is configured to determine the parsing result according to a target prompt word, the target prompt word is determined by a captured image and a preset prompt word template, and the captured image represents a key frame image captured from an image acquisition device; A generation module is used to generate a target answer text based on the target vector, wherein the target answer text is used to reply to the question text.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein the computer program can be executed by an electronic device to perform the method described in any one of claims 1 to 8.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 8 are implemented.

12. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 8 through the computer program.

Images