Large model-based inquiry processing method, device, equipment, storage medium and program product

Abstract

The present disclosure provides a large model-based inquiry processing method and device, equipment, storage medium and program product, relating to computer technology fields such as artificial intelligence interaction. The method comprises: in the camera preview window opening state of the target device, receiving the user query text sent by the target device, and acquiring the suspected image collected in the process of receiving the user query text; controlling the intent recognition process, image review process and visual reasoning process to run in parallel under the visual processing link of the large model, the intent recognition process being used to identify whether the user query text is related to vision, and the visual reasoning process being used to generate first answer information and cache; in response to identifying that the user query text is not related to vision, terminating the running of the image review process and the visual reasoning process; and controlling the text processing link of the large model running in parallel with the visual processing link to output second answer information based on the user query text. The embodiment can save computing resources.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, specifically to the field of artificial intelligence technology such as artificial intelligence interaction, and in particular to a query processing method, apparatus, electronic device, computer-readable storage medium, and computer program product based on a large model. Background Technology

[0002] Currently, artificial intelligence technology is widely used in various electronic devices. In a typical application scenario, electronic devices are equipped with audio and video capture components such as microphones and cameras. People can conveniently ask questions to these devices via voice. Servers connected to the electronic devices via a network process the questions and output answers. This intelligent questioning scenario usually requires the support of large models deployed on servers, consuming significant resources and placing considerable pressure on the servers. Summary of the Invention

[0003] This disclosure presents a query processing method, apparatus, electronic device, computer-readable storage medium, and computer program product based on a large model.

[0004] In a first aspect, embodiments of this disclosure propose a query processing method based on a large model, comprising: receiving user query text sent by the target device while the camera preview window of the target device is open, and acquiring a suspected image collected during the process of receiving the user query text; controlling the parallel execution of an intent recognition process, an image review process, and a visual reasoning process under the visual processing link of the large model; wherein, the intent recognition process is used to identify whether the user query text is visually relevant, the image review process is used to review the suspected image to obtain an image review result, and the visual reasoning process is used to generate and cache a first answer information based on the user query text and the suspected image; in response to the identification that the user query text is visually irrelevant, terminating the execution of the image review process and the visual reasoning process; controlling the text processing link of the large model to output a second answer information based on the user query text; wherein, the text processing link and the visual processing link run in parallel.

[0005] Secondly, embodiments of this disclosure propose a query processing apparatus based on a large model, comprising: a receiving unit configured to receive user query text sent by a target device when the camera preview window of the target device is open, and to acquire a suspected image collected during the process of receiving the user query text; a parallel processing unit configured to control the parallel execution of an intent recognition process, an image review process, and a visual reasoning process under the visual processing link of the large model; wherein, the intent recognition process is used to identify whether the user query text is visually relevant, the image review process is used to review the suspected image to obtain an image review result, and the visual reasoning process is used to generate and cache a first answer information based on the user query text and the suspected image; a termination unit configured to terminate the operation of the image review process and the visual reasoning process in response to the identification that the user query text is visually irrelevant; and an output unit configured to control the text processing link of the large model to output a second answer information based on the user query text; wherein, the text processing link and the visual processing link operate in parallel.

[0006] Thirdly, embodiments of this disclosure provide an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to implement the large-model-based query processing method as described in any implementation of the first aspect.

[0007] Fourthly, embodiments of this disclosure provide a non-transitory computer-readable storage medium storing computer instructions that enable a computer to implement, when executed, a large-model-based query processing method as described in any implementation of the first aspect.

[0008] Fifthly, embodiments of this disclosure provide a computer program product including a computer program that, when executed by a processor, can implement the query processing method based on a large model as described in any implementation of the first aspect.

[0009] This embodiment of the disclosure receives user query text sent by the target device while the camera preview window of the target device is open, and acquires suspected images collected during the process of receiving the user query text. It controls the parallel execution of the intent recognition process, image review process, and visual reasoning process under the visual processing link of the large model. In response to the recognition that the user query text is not visually relevant, the execution of the image review process and the visual reasoning process is terminated. It also controls the text processing link of the large model, which runs in parallel with the visual processing link, to output second answer information based on the user query text. It can terminate the execution of the image review process and the visual reasoning process in a timely manner according to the intent recognition result, thereby reducing the occupation of computing resources and saving computing resources.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is an exemplary system architecture to which this disclosure can be applied; Figure 2 A flowchart illustrating a query processing method based on a large model, provided for embodiments of this disclosure; Figure 3 A flowchart of another query processing method based on a large model provided in this disclosure embodiment; Figure 4 A flowchart illustrating yet another query processing method based on a large model provided in this disclosure embodiment; Figure 5A , Figure 5B An example diagram illustrating the visual state processing procedure in the flowchart of a query processing method based on a large model provided in this embodiment of the present disclosure; Figure 6 An example diagram of the control flow in the query processing based on a large model provided in the embodiments of this disclosure; Figure 7A A system architecture diagram for applying the query processing method based on a large model according to embodiments of this disclosure; Figure 7B A timing diagram of a query processing method based on a large model provided in an embodiment of this disclosure; Figure 8 A structural block diagram of a query processing device based on a large model provided in an embodiment of this disclosure; Figure 9 This is a schematic diagram of the structure of an electronic device suitable for performing a query processing method based on a large model, provided as an embodiment of the present disclosure. Detailed Implementation

[0012] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding; these should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description. It should be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0013] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0014] Figure 1 An exemplary system architecture 100 is shown, to which embodiments of the large-model-based query processing methods, apparatuses, electronic devices, and computer-readable storage media of this disclosure can be applied.

[0015] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0016] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various applications for enabling information communication between the terminal devices 101, 102, and 103 and server 105 can be installed. These applications include intelligent query applications and instant messaging applications.

[0017] Terminal devices 101, 102, and 103 and server 105 can be either hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices with displays, including but not limited to smartphones, tablets, laptops, and desktop computers. When terminal devices 101, 102, and 103 are software, they can be installed in the aforementioned electronic devices, and can be implemented as multiple software programs or software modules, or as a single software program or software module; no specific limitation is made here. When server 105 is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or as a single server. When server 105 is software, it can be implemented as multiple software programs or software modules, or as a single software program or software module; no specific limitation is made here.

[0018] Server 105 can provide various services through its built-in applications. Taking an intelligent query application that provides query processing services based on a large model as an example, when running this intelligent query application, Server 105 can achieve the following effects: With the target device's camera preview window open, it receives the user query text sent by the target device and acquires suspected images captured during the process of receiving the user query text; it controls the parallel execution of the intent recognition process, image review process, and visual reasoning process under the visual processing link of the large model. The intent recognition process is used to identify whether the user query text is visually relevant; the image review process is used to review suspected images to obtain image review results; and the visual reasoning process is used to generate and cache the first answer information based on the user query text and suspected images. In response to the identification that the user query text is visually irrelevant, the execution of the image review process and the visual reasoning process is terminated; the text processing link of the large model is controlled to output the second answer information based on the user query text. The text processing link and the visual processing link run in parallel.

[0019] It should be noted that server 105 can obtain user query text and suspected images from terminal devices 101, 102, and 103 via network 104.

[0020] Since query processing based on large models requires significant computing resources and power, the query processing methods based on large models provided in the subsequent embodiments of this disclosure are generally executed by a server 105 with strong computing power and abundant computing resources. Correspondingly, the query processing device based on large models is also generally located within the server 105. However, it should also be noted that when terminal devices 101, 102, and 103 also possess sufficient computing power and resources, they can also complete the aforementioned calculations performed by the server 105 through their installed intelligent query applications, thereby outputting the same results as the server 105. Especially when multiple terminal devices with different computing capabilities exist simultaneously, but the intelligent query application determines that its terminal device has strong computing power and abundant remaining computing resources, it can allow the terminal device to perform the aforementioned calculations, thereby appropriately reducing the computing pressure on the server 105. Accordingly, the query processing device based on large models can also be located within terminal devices 101, 102, and 103. In this case, the exemplary system architecture 100 may also exclude the server 105 and the network 104.

[0021] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0022] Please refer to Figure 2 , Figure 2 A flowchart of a query processing method based on a large model provided for embodiments of this disclosure, wherein process 200 includes the following steps: Step 201: With the camera preview window of the target device open, receive the user query text sent by the target device, and acquire the suspected image collected during the process of receiving the user query text.

[0023] This step is intended for the execution entity of the query processing method based on a large model (e.g., Figure 1 The server 105 shown receives user query text sent by the target device when the camera preview window of the target device is open, and acquires suspected images collected during the process of receiving user query text.

[0024] In some examples, the target device can be an electronic device with audio and video capture capabilities and network connectivity, such as a smart speaker, smart screen, home robot, in-vehicle terminal, companion terminal, etc., equipped with components such as microphone, camera, and speaker.

[0025] In some examples, the target device may be equipped with a voice recognition module. For instance, this module could be a voice assistant on the target device. Users can input questions into the target device via voice. For example, a user might say to a smart speaker, "What time is it?" After receiving the user's voice inquiry (also known as a query), the target device's voice recognition module performs voice recognition, converting the inquiry into text. The target device then uploads the text to a server over a network. In this way, the server receives the user's query text.

[0026] In other examples, the server may be configured with a speech recognition module. Users can input their questions into the target device via voice and upload the spoken questions to the server over the network. Upon receiving the user's speech, the server's speech recognition module performs speech recognition and converts the spoken question into the user's query text. Thus, the server receives the user's query text.

[0027] Step 202: Control the parallel execution of the intent recognition process, image review process, and visual reasoning process under the visual processing link of the large model. The intent recognition process is used to identify whether the user query text is visually relevant. The image review process is used to review suspected images to obtain image review results. The visual reasoning process is used to generate and cache the first answer information based on the user query text and suspected images.

[0028] Based on step 201, this step aims to have the aforementioned executing entity control the parallel execution of the intent recognition process, image review process, and visual reasoning process under the visual processing link of the large model, so as to identify whether the user query text is visually relevant, review suspected images to obtain image review results, and generate and cache the first answer information based on the user query text and suspected images in parallel.

[0029] It should be noted that the parallel operation of the intent recognition process, image review process, and visual reasoning process means that the running time of these three processes at least partially overlaps.

[0030] By controlling the parallel operation of the intent recognition process, image review process, and visual reasoning process, this step can shorten the response time to user query text, thereby improving the user experience. The response time to user query text can be the time from when the terminal sends the query to when it receives the first data packet (i.e., the initial data packet) of the response.

[0031] In one example, a suspected image refers to an image that may be related to the user's query text.

[0032] In this step, the visual reasoning process generates the first answer information based on the user's query text and the suspected image, but does not output it directly. Instead, it caches the first answer information. Whether to output the first answer information depends on the recognition results of the intent recognition process and the image review results of the image review process.

[0033] Step 203: In response to the recognition that the user query text is not visually relevant, terminate the image review process and the visual reasoning process.

[0034] Based on step 202, this step aims to have the aforementioned executing entity terminate the image review process and visual reasoning process when it recognizes that the user's query text is not visually relevant.

[0035] In some examples, if the visual reasoning process has not yet cached the first answer information, and the user query text is found to be unrelated to the visual information, the large model will no longer perform the action of caching the first answer information.

[0036] In some examples, visually relevant user query text means that the response to the user query text needs to be based on an image captured by a camera; visually irrelevant user query text means that the response to the user query text does not need to be based on an image captured by a camera.

[0037] For example, if a user inputs the voice command "How many cups are on the table?" into a smart speaker, the user's query text is also "How many cups are on the table?". This query text needs to be combined with an image captured by the smart speaker's camera to answer the question. This image must at least include the content of the space from the table surface to a certain height above the table. As another example, if a user inputs the voice command "What's the weather like in Beijing today?" into a smart speaker, the user's query text is also "What's the weather like in Beijing today?". This query text does not need to be combined with an image captured by the smart speaker's camera to answer the question.

[0038] This step reduces the consumption of computing resources and conserves computing power by promptly terminating the image review and visual reasoning processes when the user's query text is found to be unrelated to visual information.

[0039] Step 204: Control the text processing link of the large model to output the second answer information based on the user query text; wherein, the text processing link and the visual processing link run in parallel.

[0040] Building upon step 203, this step aims to have the aforementioned executing entity control the text processing link of the large model to output a second answer based on the user query text, wherein the text processing link and the visual processing link operate in parallel.

[0041] In one example, the text processing chain is used to process user query text that is not visually relevant, while the visual processing chain is used to process user query text that is visually relevant.

[0042] It should be noted that the text processing chain and the visual processing chain operate in parallel. That is, the process of generating a second answer based on the user's query text in the text processing chain can run in parallel with the aforementioned intent recognition process, image review process, and visual reasoning process. This can shorten the response time to the user's query text.

[0043] When the user query text is not visually relevant, the answer to the user query text does not require a reference image. The text processing link can generate a second answer based on the user query text and output it.

[0044] In one example, after the text processing chain generates a second answer based on the user's query text, the second answer can be cached. Once the user's query text is identified as visually irrelevant, the second answer can be retrieved from the cached area and output.

[0045] The query processing method based on a large model provided in this disclosure receives user query text sent by the target device while the camera preview window of the target device is open, and acquires suspected images collected during the process of receiving the user query text. It controls the parallel execution of the intent recognition process, image review process, and visual reasoning process under the visual processing link of the large model. In response to the recognition that the user query text is not visually relevant, the operation of the image review process and the visual reasoning process is terminated. The text processing link of the large model, which runs in parallel with the visual processing link, outputs a second answer information based on the user query text. It can terminate the operation of the image review process and the visual reasoning process in a timely manner according to the intent recognition result, thereby reducing the occupation of computing resources and saving computing resources.

[0046] The query processing method based on a large model provided in this disclosure shortens the response time to user query text and improves the user experience by controlling the parallel operation of the intent recognition process, image review process, and visual reasoning process under the visual processing link of the large model, and by enabling the visual processing link and the text processing link to run in parallel.

[0047] Please refer to Figure 3 , Figure 3 A flowchart of another query processing method based on a large model provided for embodiments of this disclosure, wherein process 300 includes the following steps: Step 301: With the camera preview window of the target device open, receive the user query text sent by the target device, and acquire the suspected image collected during the process of receiving the user query text.

[0048] This step is intended for the execution entity of the query processing method based on a large model (e.g., Figure 1 The server 105 shown receives user query text sent by the target device when the camera preview window of the target device is open, and acquires suspected images collected during the process of receiving user query text.

[0049] In some examples, before receiving the user query text sent by the target device, the large model-based query method may further include: receiving a wake-up message sent by the target device; and acquiring potential images captured during the process of receiving the user query text. This may include: identifying candidate images captured by the target device's camera during the period from receiving the wake-up message to receiving the initial data packet of the user query text as potential images, and receiving the potential images sent by the target device. In this way, images can be acquired in advance, reducing the waiting time for images during the image review and visual reasoning processes. In this case, the server can receive the user query text while simultaneously acquiring potential images. Thus, receiving the user query text and acquiring potential images are performed in parallel, which can shorten the response time to the user query text.

[0050] Step 302: Control the parallel execution of the intent recognition process, image review process, and visual reasoning process under the visual processing link of the large model. The intent recognition process is used to identify whether the user query text is visually relevant. The image review process is used to review suspected images to obtain image review results. The visual reasoning process is used to generate and cache the first answer information based on the user query text and suspected images.

[0051] Based on step 301, this step aims to control the parallel execution of the intent recognition process, image review process, and visual reasoning process under the visual processing link of the large model after receiving the user query text, so as to identify whether the user query text is visually relevant, review suspected images to obtain image review results, generate first answer information based on the user query text and suspected images, and cache and perform these processes in parallel.

[0052] In some examples, the large model for query processing includes a visual processing chain and a text processing chain. The large model inputs the received user query text into the visual processing chain and the text processing chain respectively, so that the visual processing chain and the text processing chain process the user query text in parallel, respectively obtaining a first answer and a second answer information generated for the user query text. The first answer and the second answer information are cached instead of being directly output. Then, based on the recognition results of the intent recognition process and the image review results, it is determined whether the answer information needs to be output, and if so, whether to output the first answer information or the second answer information.

[0053] In some examples, the first intent recognition model used in the intent recognition process in the visual processing chain can be a pre-trained lightweight model. This lightweight model has a short processing time and fast response, which can significantly shorten the processing time compared to the non-lightweight intent recognition model, thus obtaining intent recognition results faster.

[0054] In some examples, the text processing chain can be used to perform processing such as weather, time, Q&A, audio and video, device control, search, or general language models.

[0055] In some examples, the visual processing chain can be a chain used to acquire images and generate visual answers based on the user's query text (i.e., query), the image, and historical context.

[0056] In some examples, generating the first answer based on the user's query text and the suspected image may include: obtaining historical context related to the user's query text; and generating the first answer based on the user's query text, historical context, and suspected image. This can improve the accuracy of the first answer.

[0057] The historical context related to the user's query text can be previous query texts and / or answers to previous query texts. For example, a user makes three consecutive voice queries. In the first query, the user's query text is "Look at this," and the answer is "This is a doll." In the second query, the user's query text is "Is she pretty?" and the answer is "Very pretty." In the third query, the user's query text is "Help me design an outfit," and the answer is "You can put a tutu on her." In this example, "Look at this," "This is a doll," "Is she pretty?" and "Very pretty" constitute the historical context of the third query, "Help me design an outfit." Based on this historical context, the larger model can understand that the third query, "Help me design an outfit," is about designing an outfit for the doll, not for the user themselves.

[0058] In some examples, obtaining historical context related to the user's query text includes: in response to the user's query text being converted from the user's speech using speech-to-text technology, extracting historical question-and-answer corpus related to the user's speech from the historical question-and-answer database of the corresponding user stored on the target device; and using the historical question-and-answer corpus as historical context related to the user's query text. In this way, the historical context related to the user's query text can be accurately obtained, thereby improving the accuracy of the generated answer information.

[0059] Step 303: In response to the recognition that the user query text is not visually relevant, terminate the image review process and the visual reasoning process.

[0060] Building upon step 302, this step aims to promptly terminate the image review process and the visual reasoning process when the user's query text is found to be unrelated to visual information.

[0061] Step 304: Control the text processing link of the large model to output the second answer information based on the user query text; wherein, the text processing link and the visual processing link run in parallel.

[0062] Building upon step 303, this step aims to identify situations where the user query text is unrelated to visual information, and then control the text processing chain of the large model to output a second answer based on the user query text.

[0063] Steps 301-304 above are similar to... Figure 2 The steps 201-204 shown are the same. For the same parts, please refer to the corresponding parts of the previous embodiment. They will not be repeated here.

[0064] In some examples, before the text processing chain of the control large model outputs the second answer information based on the user query text, the method further includes: sending the recognition result of the user query text identified by the intent recognition process as visually irrelevant to the access control module in the text processing chain; controlling the access control module to verify whether the user query text is visually irrelevant, and after the access control module obtains a corrective response indicating that the user query text is visually relevant, recording an error for the intent recognition process that the recognition result is incorrect. In this way, the parameters of the model used by the intent recognition process can be adjusted based on the error record of the incorrect recognition result, improving the accuracy of the intent recognition result.

[0065] In some examples, the access control module in the text processing chain can be the aforementioned second intent recognition model.

[0066] In some examples, before the text processing link of the large model outputs the second answer information based on the user query text, the method further includes: sending the recognition result of the user query text being visually irrelevant, identified by the intent recognition process, to the access control module in the text processing link; controlling the access control module to verify whether the user query text is visually irrelevant; and controlling the text processing link of the large model to output the second answer information based on the user query text, including: after the access control module verifies and obtains a confirmation response that the user query text is visually irrelevant, controlling the text processing link of the large model to output the second answer information. This can improve the accuracy of the text processing link's output.

[0067] Step 305: In response to not receiving user query text from the target device within a preset time period, send a preview window closing notification to the target device so that the target device closes the camera preview window according to the preview window closing notification.

[0068] Building upon step 304, this step aims to send a preview window closing notification to the target device in response to the failure to receive a user query text from the target device within a preset time period, so that the target device can promptly close the camera preview window based on the notification. For example, the preset time period can be 1 minute.

[0069] If the server does not receive a user query text from the target device within the preset time period, it indicates that the user of the target device has not used the target device within the preset time period, and the probability of the user continuing to use the target device is very small. Therefore, this step, where the server actively closes the camera preview window of the target device when the user has not used the target device for an extended period, can save the target device's power and thus increase the remaining usable time of the target device.

[0070] In some examples, step 305 can be implemented as follows: the command sent to the target device to open the camera preview window includes configuration information, which includes a preset duration. The client on the target device starts a timer based on the configuration information to monitor the duration the user has not used the camera preview window. If the timer reaches the preset time, the camera preview window on the target device is closed.

[0071] In some examples, step 305 can also be replaced by: in response to not receiving user query text from the target device within a preset time period, sending a preview window closing request to the target device, and in response to receiving a response message indicating that the preview window should be closed, sending a preview window closing notification to the target device. Thus, when the query is unrelated to vision, the server can close the target device's camera preview window according to the user's wishes, thereby saving the target device's power and increasing its remaining usable time.

[0072] In some examples, the intent recognition process can also be used to identify whether the user's query text contains an intent to exit the camera preview window. In this case, the query processing method based on the large model can further include: in response to receiving the user's query text within a preset time period and recognizing through the intent recognition process that the user's query text contains an intent to exit the camera preview window, sending a preview window closing notification to the target device, so that the target device closes the camera preview window according to the preview window closing notification. For example, when the user's query text is "Don't look anymore" or "Close the camera," the recognition result of the intent recognition process is: the user's query text contains an intent to exit the camera preview window. This embodiment can exit the camera preview window in a timely manner according to the user's intent.

[0073] The query processing method based on a large model provided in this disclosure receives a wake-up message and user query text from the target device while the camera preview window of the target device is open. In response to receiving the wake-up message, the method controls the target device to acquire a suspected image and receives the suspected image sent by the target device. It controls the intention recognition process, image review process, and visual reasoning process to run in parallel under the visual processing link of the large model. In response to recognizing that the user query text is unrelated to vision, the method terminates the operation of the image review process and the visual reasoning process. It also controls the text processing link of the large model, which runs in parallel with the visual processing link, to output a second answer message based on the user query text and sends a preview window closing notification to the target device so that the target device can close the camera preview window according to the preview window closing notification. This method can not only terminate the operation of the image review process and the visual reasoning process in a timely manner according to the intention recognition result, thereby reducing the occupation of computing resources and saving computing resources, but also close the camera preview window of the target device in a timely manner, saving the power of the target device and increasing the remaining usable time of the target device.

[0074] Please refer to Figure 4 , Figure 4 A flowchart of another query processing method based on a large model provided in this disclosure embodiment, wherein process 400 includes the following steps: Step 401: With the camera preview window of the target device open, receive the user query text sent by the target device, and determine the candidate images captured by the camera of the target device during the suspected period from the start of receiving the initial data packet constituting the user query text to the receipt of the end data packet of the user query text as suspected images, and receive the suspected images sent by the target device.

[0075] This step is intended for the execution entity of the query processing method based on a large model (e.g., Figure 1 The server 105 shown receives user query text sent by the target device when the camera preview window of the target device is open. It then identifies candidate images captured by the target device's camera during the suspected period from the start of receiving the initial data packet constituting the user query text to the receipt of the final data packet of the user query text, and receives the suspected images sent by the target device. This allows for advance image acquisition, shortening the response time to the user query text.

[0076] In some examples, the target device's camera preview window can be opened upon first hitting a visual intent (hitting a visual intent means the user's query text is visually relevant). For instance, when the target device's camera preview window is closed, the server receives the user's query text from the target device and identifies it as visually relevant through the second intent recognition model of the text processing link. At this point, the server can detect whether the large model's visual processing link is open, confirm whether the user has enabled image privacy authorization, and confirm whether the target device's camera is on and usable. If the server detects that the current large model's visual processing link is not open, it opens the visual processing link; if the server confirms that the current user has not enabled image privacy authorization, it initiates the authorization process, requests image privacy authorization from the user, and marks the user as having enabled image privacy authorization after the user approves the request; if the server detects that the target device's camera is closed, it requests the user to open the camera, and opens the target device's camera after the user approves the request. Thus, when the large model's visual processing link is open, the user has enabled image privacy authorization, and the target device's camera is on and usable, the server can confirm that the target device's camera preview window is open; this state can be referred to as the display visual state. It should be noted that the second intent recognition model in the text processing chain is different from the first intent recognition model that performs the intent recognition process in the visual processing chain. The former is a more complex, larger model, while the latter is a lightweight, smaller model. The second intent recognition model in the text processing chain can accurately identify whether the user's query text is visually relevant.

[0077] In some examples, identifying candidate images captured by the target device's camera during the period from the start of receiving the initial data packet constituting the user's query text to the receipt of the final data packet of the user's query text as potential images can include: in response to the fact that the candidate images captured by the target device's camera during the potential period are at least three images arranged in chronological order, identifying at least one candidate image located at the beginning, middle, and end of the potential period as a potential image; wherein multiple potential images are arranged in chronological order. This allows for accurate location of the image corresponding to the user's query text, thereby improving the accuracy of the visual response.

[0078] Step 402: Control the parallel execution of the intent recognition process, image review process, and visual reasoning process under the visual processing link of the large model. The intent recognition process is used to identify whether the user query text is visually relevant. The image review process is used to review suspected images to obtain image review results. The visual reasoning process is used to generate and cache the first answer information based on the user query text and suspected images.

[0079] Building upon step 401, this step aims to control the parallel execution of the intent recognition process, image review process, and visual reasoning process within the visual processing chain of the large model.

[0080] Step 403: In response to the recognition that the user's query text is visually relevant and the image review result indicates that the review has been passed, output the cached first answer information.

[0081] Building upon step 402, this step aims to identify situations where the user's query text is visually relevant and the image review result indicates approval. In this case, the cached first response information is output and sent to the target device. This prevents the visual result from being played prematurely before the image review is complete.

[0082] In some examples, step 403 can be replaced by: in response to recognizing that the user's query text is visually relevant and the image review result indicates that the review has failed, clearing the cached first and second answer information. That is, when it is recognized that the user's query text is visually relevant and the image review result indicates that the review has failed, neither the first nor the second answer information is output. In this case, the server can also output rejection reason information, which indicates why the server does not output the answer information. For example, the rejection reason information can be "image review failed," "image contains non-compliant information," etc. This embodiment can clear the cache in a timely manner.

[0083] The query processing method based on a large model provided in this disclosure receives user query text sent by the target device while the camera preview window of the target device is open. After receiving the first data packet of the user query text but before receiving the complete information of the user query text, the method controls the target device to acquire a suspected image and receive the suspected image sent by the target device. The method controls the parallel execution of the intent recognition process, image review process, and visual reasoning process under the visual processing link of the large model. In response to the recognition that the user query text is visually related and the image review result indicates that the review has passed, the method outputs the cached first answer information, which can shorten the response time to user query text and improve the user experience.

[0084] Assuming the processing time of the text processing chain is T_m, the processing time of generating the first answer information (which can be the first valid content in the answer information generated for the user query text) in the visual reasoning process is T_v, the processing time of the intent recognition process is T_i, the processing time of the image review process is T_c, the processing time of the image acquisition process is T_p, and the processing time of caching, access control policy calculation, forwarding, etc. is T_o, then when the first answer information is output through process 400, the response time of the user query text is T = T_p + max(T_i, T_c, T_v) + T_o. This is significantly shorter than the response time of the user query text (T_p + T_i + T_c + T_v) in the case of serial processing (intent recognition process, image review process, and visual reasoning process run serially). The effect of shortening the time is even more significant when T_i, T_c, and T_v are all non-negligible.

[0085] To enhance understanding, this disclosure also provides a specific implementation scheme in conjunction with a particular application scenario. In this scheme, the process of a query processing method based on a large model provided in this disclosure embodiment may include the following steps: Step 1: Receive wake-up signal or user input.

[0086] In one example, wake-up signals or user input can be received by the user input and speech recognition module in the large model.

[0087] In this step, the server receives the user's wake-up signal and voice input, and obtains the user's query text through voice recognition.

[0088] The server can further determine the query request context for this round of queries. This query request context may include: a request identifier; the user query text; the intent set identified by the text processing chain; device and user identifiers; whether the camera is on; the visual capability on / off status (the visual capability on / off status is the aforementioned visual processing chain switch); image privacy authorization status; whether the camera preview window is continuously displayed; the current screen or application status; the risk level; and short-term conversation context (i.e., historical context).

[0089] Specifically, if the intent set identified by the text processing link includes the visual intent hit identifier `vision_agent.start`, it indicates that the intent identification result of the text processing link is visually relevant to the user's query text in this round. If the intent set identified by the text processing link does not include the visual intent hit identifier `vision_agent.start`, it indicates that the intent identification result of the text processing link is visually irrelevant to the user's query text in this round.

[0090] Step 2: Determine the timing of taking a photo based on the visual state.

[0091] In one example, the camera and preview control module in the large model can determine the timing of taking a picture based on the visual state.

[0092] The timing of taking a photo is the time when an image is captured.

[0093] Before performing image acquisition, the server confirms that the user has enabled image privacy authorization and that the device's camera is on or available. If the user has not completed authorization, the vision capability switch is off, or the camera is off, the server will not proceed with the pre-acquisition and visual inference process. Instead, it will trigger a vision capability activation guide, a camera off prompt, or output preset information indicating that the conditions for image acquisition are not currently met.

[0094] When the second intent recognition model in the text processing chain first determines that the query is related to vision, the server controls the target device to open the camera preview window and start the image acquisition process.

[0095] When the target device is already in a visual display state, to reduce waiting time, the server can directly issue a photo-taking command when the user wakes up, without waiting for the complete query and the current intent recognition process to complete. The captured image can be associated with the current query processing through a request identifier.

[0096] It should be noted that taking photos in advance only means that the image acquisition time is moved forward to the wake-up or voice recognition stage, provided that the user has entered visual interaction, the preview window is continuously displayed, and the authorization is valid. This embodiment does not take photos in the background without authorization or visual prompts.

[0097] It should be noted that the timing of taking photos in the above example is not unique. Image acquisition can at least partially overlap with other processing procedures.

[0098] In some examples, Image acquisition (i.e., taking a picture) can be performed at any of the following times: When the user wakes up; When speech recognition begins; When the complete user query text is obtained; After the second intent recognition model in the text processing chain identifies the visual intent; When VisionAgent requests an image; When selecting the latest keyframe relevant to the current query from the short-term video cache.

[0099] Step 3: The outer layer initiates parallel processing of the text processing link and the visual processing link.

[0100] After receiving a complete user query request, the server can synchronously send the query request to the text processing link and the visual processing link respectively, so that the processing of the two links is at least partially parallel.

[0101] In this example, the visual processing chain in the large model can include agents: SenseAgent and VisionGateAgent. VisionGateAgent can internally include agents: VisionAgent, CensorAgent, and ExitDecisionAgent. SenseAgent can create a rendering agent, RenderAgent.

[0102] In some examples, the text processing pipeline can perform tasks such as weather forecasting, searching, Q&A, or device control in advance. The vision processing pipeline, on the other hand, waits for or retrieves the images associated with the current query request and begins vision-related reasoning.

[0103] It should be noted that the inference results of the visual processing link and the text processing link (i.e., the aforementioned first and second answer information) will not both be output to the user. This step utilizes parallel execution to overlap the intent recognition process and visual inference process of the visual processing link with the inference process of the text processing link, thereby reducing time consumption.

[0104] Step 4: Verify the device and request status.

[0105] After receiving a query request, the SenseAgent agent in the vision processing chain can perform the following checks: 1. Initialize request logs and device context; 2. Check if the intent set identified by the text processing chain contains vision_agent.start; 3. If a visual intent is hit (if the intent set identified by the text processing link contains vision_agent.start, it means a visual intent has been hit), mark this round of processing as displaying visual interaction and check the visual capability switch; 4. If the visual capability is disabled, trigger the device command to enable the visual capability and end the current processing round; 5. Check image privacy authorization, whether the camera is turned on, and whether the device supports this round of image acquisition; 6. Maintain the preview window duration while displaying visual status; 7. Check the risk level of the request; 8. Waiting for the image associated with this round of query requests from the resource manager.

[0106] If there is no authorization, the camera is turned off, the image times out, the image type is incorrect, or the VisionAgent used for processing the vision processing link fails to be created, the exception handling process will be initiated.

[0107] For example, if the collected images are uploaded to the server slowly due to network problems, it will cause image timeout; if some data is lost due to network transmission problems, it will cause image type errors; if the code fails to run due to server problems (such as insufficient memory), it will cause the creation of the VisionAgent to fail.

[0108] Step 5: Construct the input information for the main visual response to generate the intelligent agent VisionAgent.

[0109] The following content can be constructed into multimodal user messages and used as input information for VisionAgent: The current user is querying a text query; The Base64 data and MIME (Multipurpose Internet Mail Extensions) type of the currently acquired image; Recovering historical messages from short conversation contexts.

[0110] Image data is only transmitted as multimodal input to the current query request during the process of connecting the edge and cloud, server-side memory cache, image review service, and visual processing; the original image is not persisted to disk. The corresponding image cache should be released or invalidated after the request completes, times out, exits the visual state, or is rejected by access control.

[0111] In some examples, the visual processing link visual response (i.e., the first response information) can be generated in a streaming mode.

[0112] In some examples, if image retrieval times out—that is, if the VisionAgent fails to retrieve an image from the resource manager within a preset time—the current round of visual inference is stopped and a timeout handling process is executed. This exception occurs before the VisionGateAgent is created, therefore it is not a GateDecision exception.

[0113] Step 6: Start the parallel tasks within the vision processing chain.

[0114] The VisionGateAgent visual access control orchestration module creates cancelable sub-contexts and launches them in parallel on the same input: 1. VisionAgent, the main visual response generator: generates visual main events (i.e., the aforementioned first response information) or message streams. 2. Image-based access control agent CensorAgent: Waits for the asynchronous result of the image review service; 3. ExitDecisionAgent: Calls the lightweight multimodal model (i.e., the intent recognition module in the visual processing chain) to determine whether the current request should be passed, exited, or transferred, i.e., performs the exit / transfer judgment.

[0115] The three tasks mentioned above do not form a sequential dependency of "reasoning can only be done after the review is completed" or "reasoning can only be done after the intention is completed".

[0116] Among them, ExitDecisionAgent is the first intent recognition model within the visual processing chain.

[0117] It should be noted that ExitDecisionAgent is not a generalized visual dependency confidence classifier. In one example, the outcome can be controlled through tool actions: Exit or handover tool not triggered: Allow=true, Reason=pass; Trigger the exit tool: Allow=false, Reason=exit; Trigger the transfer tool: Allow=false, Reason=transfer: target Agent.

[0118] In some examples, ExitDecisionAgent may internally include an internal small model Agent, AnswerWithVisionTools, TransferAgentTools, and ExitAgentTools. The internal small model Agent determines whether the cause is pass, exit, or transfer; AnswerWithVisionTools generates a pass signal when the cause is pass; TransferAgentTools generates transfer:general or other handover signals when the cause is transfer; and ExitAgentTools generates an exit signal when the cause is exit.

[0119] Step 7: Cache the main visual events.

[0120] The main visual event is the visual response event generated by VisionAgent. VisionAgent can send the main visual event to VisionGateAgent through a separate channel. Before the gate is fully completed, the main visual event is stored in an internal buffer and is not handed over to the downstream rendering agent RenderAgent.

[0121] If the main visual event contains a message stream handle, the system can wrap the message stream to record the actual generation time of the first non-empty fragment of the visual model (VisionAgent). This time is recorded separately from the gate release time to distinguish between model inference time and gate waiting time.

[0122] The "caching" here includes at least storing events generated by the main agent. In other implementations, text fragments can also be cached, downstream consumer stream handles can be delayed, or backpressure can be created through bounded queues.

[0123] Step 8: Collect and aggregate access control decisions (also known as access control conclusions).

[0124] This example includes two access control systems: an image review agent (CensorAgent) and an output decision agent (ExitDecisionAgent). After each access control is completed, the access control conclusion (GateDecision) is written to the access control result channel. The CensorAgent's GateDecision is `image_censor`, where `image_censor` can be one of `pass`, `reject`, `disabled`, or `timeout`. The ExitDecisionAgent's GateDecision is `exit_decision`, where `exit_decision` can be one of `pass`, `exit`, or `transfer`.

[0125] In some examples, the fields included in GateDecision and their meanings can be seen in Table 1 below: Table 1

[0126] VisionGateAgent records the number of access control devices pending completion and executes the access control policy GatePolicy after all registered access control devices have returned.

[0127] In some examples, the access control policy GatePolicy enforced by VisionGateAgent can be: 1. If all Access Control gates have a true value in their GateDecision class, the entire gate is allowed. 2. If the Allow setting in the GateDecision property of any access control system is false, the entire system will be rejected. 3. When multiple access control systems simultaneously refuse access, the final reason for refusal will be selected based on the priority of the registration source.

[0128] In one example, the registration priority can be shown in Table 2 below: Table 2

[0129] According to Table 1, when both image review and transfer or exit are rejected, image review rejection takes precedence.

[0130] In this example, the priority policy can be executed after all departmental rejection conclusions are received, rather than immediately short-circuiting upon arbitrary rejection. In other examples, early short-circuiting is also possible, i.e., short-circuiting immediately upon arbitrary rejection.

[0131] In some examples, after the access control is rejected, the visual context can be canceled, the reading events can be stopped, and the stream can be closed, which can reduce the continued occupation of model computation, network connection, and coroutine resources.

[0132] Step 9: Allow cached visual master events when all doors are restricted.

[0133] When the access control policy returns permission, VisionGateAgent performs the following operations: 1. Record the waiting time for access control; 2. Switch the status to allowed; 3. Send the previously cached main visual events in the original order of the VisionAgent events; 4. Clear the buffer zone; 5. Send subsequent main visual events directly to the downstream RenderAgent; 6. The output adaptation layer continues to complete text aggregation, task updates, and voice output.

[0134] The visual first packet waiting time is determined by the visual model generation speed and the slowest access control time, and is close to the larger of the two, rather than the sum of the two.

[0135] Step 10: Cancel the main visual event when the access control system refuses access.

[0136] When the access control policy returns a rejection, VisionGateAgent performs the following operations: 1. Do not allow cached main visual events to proceed; 2. Generate an AgentEvent that only carries control actions; 3. Place the selected GateDecision into a CustomizedAction; 4. Cancel the sub-context used by VisionAgent; 5. Stop the output of VisionGateAgent in this round.

[0137] This structured rejection event is separated from the regular visual text to prevent the reason for rejection from being mistakenly played as a visual response.

[0138] Step 11: SenseAgent performs the action based on the reason for rejection.

[0139] After receiving a GateDecision with Allow=false, SenseAgent performs the following actions based on the Reason and the displayed visual state.

[0140] Scenario 1: Pass

[0141] With all access controls enabled, the visual stream (i.e., the main visual event) is output normally through the RenderAgent and the output adapter module Adaptor.

[0142] Scenario 2: Exit

[0143] The system disables the camera preview or clears the visual state, stops listening, and plays an exit confirmation message.

[0144] For example, the user inputs the voice command "Don't look anymore" or "Turn off the camera." The ExitDecisionAgent triggers ExitAgentTools, generating Reason=exit. The VisionGateAgent blocks the visual response; the SenseAgent closes the preview window or clears the visual state and plays an exit confirmation message.

[0145] Scenario 3: Transfer occurs under the displayed visual state.

[0146] The visual state displayed here is the state after the second intent recognition model in the text processing chain has generated the visual intent vision_agent.start and opened the camera preview window.

[0147] If ExitDecisionAgent determines that the current request should be transferred in this state, the system switches to the text processing link and outputs the text response (i.e., the second response information) generated by the text processing link.

[0148] For example, if the target device's camera preview window is already open, and the user submits a query that is irrelevant to the image or unsuitable for VisionAgent to answer, ExitDecisionAgent calls TransferAgentTools to generate transfer:general.

[0149] Furthermore, VisionGateAgent blocks visual responses and cancels the visual task. SenseAgent detects that it is currently in a visual display state and outputs the text response information from the text processing chain.

[0150] Based on scenarios two and three, it can be seen that in this embodiment, the preview window will be closed when the reason is "exit" and a clear prompt message will be output when the reason is "transfer" in the visual display state. This can avoid the inconsistency between the target device screen state and the result heard by the user, thereby further improving the user experience.

[0151] Scenario 4: Transfer occurs in a non-display visual state.

[0152] When the query request is not in the display visual state, the visual processing link generates an upstream fallback task through AgentFallBack(reason), transmits the transfer signal to the upstream module, and the upstream module determines the text response information of the output text processing link based on the transfer signal.

[0153] Scenario 5: Image rejected during review

[0154] If the query request is in the display visual state, stop listening, write the final prompt message, and play a prompt message to indicate that the image is safe; If the query request is in a non-visual state, AgentFallBack is invoked to return the rejection reason to the upstream for processing.

[0155] In some examples, VisionAgent may have already generated candidate visual events (i.e., visual responses), but these events remain within the visual processing chain. CensorAgent returns Allow=false, Reason=reject. At this point, the system does not release the visual response, cancels the visual task, and plays a security warning in the displayed visual state. This prevents risky content from being output when the image review conclusion lags behind the first response data packet of the large model.

[0156] Step 12: Handling access control anomalies.

[0157] In some examples, the following strategies can be used for different exceptions: If the exception is: CensorAgent is not enabled, then return: Allow=true, Reason=disabled (not enabled); If the exception is: image review timeout, empty result, or review service error, then return: Allow=true, Reason=timeout_or_err: ; If the exception is: ExitDecisionAgent model initialization, Prompt (hint) retrieval, Agent creation, or model event failure, then retain the default Allow=true and Reason=pass. If VisionAgent fails to create or returns an error during operation, the system logs the exception, stops visual response output, and displays a prompt message. If the upstream parallel text processing chain has the capacity to handle the additional processing, the modules in that chain can perform further processing.

[0158] If the exception is: the access control returns an object whose type does not match the GateDecision, then propagate the error upwards and cancel the vision task; If the exception is: VisionAgent itself returns an error, then the error is propagated upwards and the vision task is canceled.

[0159] In some examples, image review timeouts or ExitDecisionAgent exceptions are allowed to continue processing by default, i.e., fail-open, to ensure the availability of the vision service and prevent access control service failures from making the entire vision capability unavailable. This policy can be adjusted based on the risk level of the user's query text.

[0160] In some examples, in high-risk scenarios such as medical, financial, or child protection, failure-close is used when image review times out or ExitDecisionAgent is abnormal, meaning that visual output is prohibited when the access control is abnormal.

[0161] Step 13: End of query request and handling of late results.

[0162] VisionGateAgent uses a cancelable context notification to stop the main vision task and the access control task. The event is sent while simultaneously listening for a context cancellation signal; the wrapper for the vision stream closes the original stream when the downstream is shut down.

[0163] For external model calls that cannot be physically terminated, late results can be invalidated by stopping reading, closing the stream, or requesting an end sign.

[0164] The above solution does not require waiting for the response information from both the text processing link and the visual result processing link to be ready before determining which link's response result should be output.

[0165] In some examples, the above scheme can be implemented in two layers of parallelism. The first layer of parallelism involves the text processing and vision processing chains, acting as upstream modules, operating in parallel. The second layer of parallelism involves the VisionAgent, CensorAgent, and ExitDecisionAgent within the vision processing chain operating in parallel. The first layer of parallelism reduces serial waiting between the text processing and vision processing chains; the second layer of parallelism reduces serial waiting between visual inference (performed by VisionAgent) and gate control decisions (performed by CensorAgent and ExitDecisionAgent). VisionGateAgent ensures that the second layer of parallelism still has a clearly defined output boundary.

[0166] The first layer of parallelism is the outer layer of parallelism. In some examples, the visual processing chain and the text processing chain can adopt any of the following parallel methods: Start up completely simultaneously; Set up temporary off-peak times; Whether to immediately initiate the visual processing link depends on the displayed visual status; Adjust the startup timing based on historical visual request ratios, device load, or network quality.

[0167] According to the above scheme, although VisionAgent can perform visual reasoning in advance to obtain visual answers, the visual answers cannot reach RenderAgent before the access control allows it, thus balancing reasoning speed and content security.

[0168] In some examples, the vision processing chain can adopt any of the following models: Cloud-based multimodal large models; End-side visual language model; A combination of Optical Character Recognition (OCR), object detection, and language models; The model can be either developed in-house or by a third party.

[0169] In some examples, ExitDecisionAgent can be replaced with: Text or multimodal classification models; Rules and keywords; Small parameter language model; Decision trees or other classifiers; Combination of rules and models.

[0170] In some examples, regarding the number and policies of access control systems, in addition to image verification and exit / transfer decisions, at least one of the following can be added: Privacy check; Account or device permissions check; Inspections for the protection of minors; Regional compliance checks; Output quality or factual checks.

[0171] In some examples, access control aggregation can employ methods such as full allow, risk weighting, tiered strategies, or high-risk denial short-circuiting.

[0172] In some examples, the visual responses generated by the visual processing chain can be isolated in either of the following ways: Cache AgentEvent; Cache text fragments; Delaying the delivery of MessageStream downstream; Pausing to read upstream flow creates back pressure; Write to a memory queue, in-process cache, or a temporary transfer buffer with lifecycle constraints.

[0173] In some examples, regarding privacy and data lifecycle constraints, the following privacy protection constraints can be adopted in different deployment methods: Before image acquisition, check the user's image privacy authorization, visual capability switch, and camera on / off status; Only when the visual state is displayed, when the user actively requests a visual response, or when there is an equivalent visual interaction, should the forward-moving photo opportunity be taken. Bind the image to the request identifier, limiting its use to this round of interaction; The original image is only used temporarily in edge-cloud transmission, server-side memory caching, image review, and visual inference calls. Do not write the original image to a persistent file, database, or long-term object store; Release or invalidate the image cache when the request ends, times out, the user exits the visual state, the review is rejected, or the task is canceled.

[0174] In some examples, any of the following strategies can be used when access control malfunctions: fail-open; fail-close; Retry; Activate the backup access control system; Degrade to text processing pipeline processing or manual processing; Strategies are dynamically selected based on risk levels.

[0175] In some examples, for tasks that are not selected or rejected, any of the following termination methods can be used to terminate them: Cancel context; Close the model stream; Stop receiving subsequent events; Late marking invalidates the result; Lower the priority of background tasks; Only retain the necessary performance logs.

[0176] Figure 5A , Figure 5B This diagram illustrates an example of the visual state handling process in a query processing method based on a large model, provided in an embodiment of this disclosure. Please refer to... Figure 5A and Figure 5B The visual status handling process 500 includes the following steps: Step 501: SenseAgent receives the query request (handleIntentRequest).

[0177] Step 502: Determine whether the intent set identified by the text processing link contains the visual intent hit identifier vision_agent.start. If yes, proceed to step 503; otherwise, proceed to step 506.

[0178] In this step, if the intent set identified by the text processing link contains vision_agent.start, it means that the intent set contains visual intent; if the intent set identified by the text processing link does not contain vision_agent.start, it means that the intent set does not contain visual intent.

[0179] Step 503: Set the s.isVisionStart flag to true.

[0180] In some examples, s.isVisionStart is used to indicate the display vision state. s.isVisionStart=true indicates the display vision state, and s.isVisionStart=false indicates the non-display vision state (i.e., not in the display vision state).

[0181] Step 504: Determine whether the vision ability switch is turned on. If yes, proceed to step 505; otherwise, proceed to step 507.

[0182] Step 505: Determine that the value of the listener flag SetNeedListen is false, issue the command to enable deeplink vision capability, and end.

[0183] Step 506: Set the s.isVisionStart flag to false.

[0184] Step 507: Determine if the camera is on. If yes, proceed to step 512; otherwise, proceed to step 508.

[0185] Step 508: HandleFallback is executed because the camera is closed (reason = camera is closed).

[0186] Step 509: Determine if s.isVisionStart? If yes, proceed to step 510; otherwise, proceed to step 512.

[0187] Step 510: Display a message indicating that the camera is off (Say (CAMERA_CLOSE)), then end.

[0188] Step 511: Return to upstream (AgentFallback (reason)), end.

[0189] Step 512: Determine if s.isVisionStart? If yes, proceed to step 513; otherwise, proceed to step 514.

[0190] Step 513: Open and maintain the camera preview window (ControlCameraPreview(ture, keepTime)), then proceed to step 515.

[0191] Step 514: Do not actively open the camera preview window, proceed to step 515.

[0192] Step 515: Determine if the risk level is greater than 0 (RiskLevel>0?). If yes, proceed to step 516; otherwise, proceed to step 517.

[0193] Step 516: HandleFallback process, displaying a prompt message indicating a review rollback (Say (CENSOR_FALLBACK)) or returning to the upstream (AgentFallback), then end.

[0194] Step 517: Wait for the image in the image cache (ImageCache), set the timeout to 2 seconds. If the wait times out, the type is incorrect, or the Agent creation fails, proceed to step 518; if the image is successfully obtained, proceed to step 522.

[0195] Step 518: HandleFallback process, displaying a message indicating that the image waiting timeout (TIMEOUT_FALLBACK) or a message indicating that the camera is off (CAMERA_CLOSE).

[0196] Step 519: Determine if s.isVisionStart? If yes, proceed to step 520; otherwise, proceed to step 512.

[0197] Step 520: Play the preset visual scene prompts and end.

[0198] Step 521: Return to upstream (AgentFallback (reason)), end.

[0199] Step 522: Construct the intelligent agents VisionAgent, CensorAgent, and ExitDecisionAgent, and enter the internal processing flow of VisionGateAgent.

[0200] Step 523: Determine whether VisionGateAgent returns a deny gate decision (GateDecision). If not, proceed to step 524. If so, proceed to step 525, step 527, or step 532 according to the reason.

[0201] Step 524: RenderAgent outputs a visual response, plays the visual result, and ends.

[0202] Step 525: If the reason for rejection is exit (Reason=exit), proceed with the exit process (handleExit).

[0203] Step 526: Close the camera preview window and / or clear the display visual status to finish.

[0204] Step 527: If the rejection reason includes "transfer" (the reason includes "transfer"), proceed with the transfer process.

[0205] Step 528: Perform fallback processing (handleFallback, Reason=transfer: target Agent, TTS=EXIT_FALLBACK), where TTS stands for Text-to-Speech.

[0206] Step 529: Determine if s.isVisionStart? If yes, proceed to step 530; otherwise, proceed to step 531.

[0207] Step 530: Without switching to the text processing link, play the preset visual scene prompt (EXIT_FALLBACK) to end.

[0208] Step 531: Return to upstream (AgentFallback(transfer: target Agent)), end.

[0209] Step 532: Determine if s.isVisionStart? If yes, proceed to step 533; otherwise, proceed to step 534.

[0210] Step 533: Stop listening, play the security message (SetNeedListen(false), Say(BlockMessage)), and end.

[0211] Step 534: Return AgentFallback (decision.Reason) to the upstream agent, and the process ends.

[0212] In some examples, the data flow of a query processing method based on a large model can be: (1) Receive user voice input; (2) The user’s voice input is processed by Automatic Speech Recognition (ASR) to obtain the user’s query text. (3) The user query text is input into the second intent recognition model of the text processing link, and the second intent recognition model of the text processing link outputs the visual intent (i.e., determines that the user query text in this round is visually related). (4) Based on visual intent, enable the target device to enter the camera preview window open state or confirm that the target device's camera preview window is open; (5) Perform image acquisition and cache the acquired images; (6) Construct the input information of VisionAgent using “user query text + collected image + historical context of user query text”; (7) Input the user query text + the collected image + the historical context of the user query text into VisionAgent, and VisionAgent will output the main visual event; (8) Cache main visual events; (9) Confirm that all access controls are enabled; (10) Input the cached visual main event into RenderAgent for rendering; (11) After rendering, the data enters the streaming output adapter; (12) The Adaptor outputs the corresponding audio of the TTS-processed response text, or outputs the response text through the interface.

[0213] The image acquisition and caching processes are constrained by image privacy authorization, camera on / off status, and display visual status. Images are bound to the current interaction through request identifiers and transmitted as temporary inputs in the processing chain. The system does not save the original images to persistent storage; the logs can record necessary metadata such as request identifiers, time consumption, image size, or review results, but do not record the original image content.

[0214] In some examples, the control flow in a query processing procedure based on a large model can be as follows: Figure 6 As shown. Please see below. Figure 6 The access control policy (GatePolicy) determines whether to output "allow" or "deny" based on the image review results and the exit / transfer judgment results. If the access control policy outputs "allow", the large model outputs the main visual event. If the access control policy outputs "deny", the large model propagates the access control conclusion (GateDecision) and then executes "exit", "fallback", or security prompt.

[0215] To further enhance understanding, this section will provide a more detailed explanation of the query processing based on a large model in this embodiment, taking into account the specific application scenarios of smart speakers.

[0216] In this example, initially, the smart speaker's camera preview window is not open. The user inputs the voice command, "Help me see what this is." In this case, the second intent recognition model in the text processing link outputs a visual intent based on "Help me see what this is," and the server controls the smart speaker to open its camera preview window based on this visual intent. Then, the server controls the smart speaker to capture images and initiates the main visual response generation agent, VisionAgent. Within the visual processing link, the intent recognition process (i.e., exit / transfer judgment), image review process, and visual reasoning process (for generating a visual response) are performed in parallel. If both the image review result and the exit / transfer judgment result allow, the visual response is released (i.e., output), and the visual response is then read aloud.

[0217] Next, while the smart speaker's camera preview window remains open, the user, after inputting the voice command "Help me see what this is," inputs a wake word and continues with the question "What's that next to me?" Regarding the question "What's that next to me?", during the wake-up phase upon receiving the wake word, the server preemptively sends a photo-taking command to the smart speaker. Thus, image acquisition begins or is completed before the server receives the complete query text. After receiving the full query text "What's that next to me?", the server initiates parallel processing of the text processing and visual processing links. Within the visual processing link, the intent recognition process (i.e., exit / transfer judgment), image review process, and visual reasoning process (for generating a visual response) are performed in parallel. If both the image review result and the exit / transfer judgment result allow, the first data packet of the visual response is quickly released. This overlaps the time spent on photo taking, visual reasoning, image review, and exit / transfer judgment, thereby shortening the initial response time for the visual query.

[0218] Next, while the smart speaker's camera preview window remains open, after the user asks "What's that next to me?" and receives a visual response, they continue by inputting the voice message "What time is it?". Before receiving "What time is it?", the server preemptively controls the smart speaker to capture an image. After receiving "What time is it?", it initiates the text processing and visual processing links in parallel. Within the visual processing link, the intent recognition process (i.e., exit / transfer judgment), image review process, and visual reasoning process (used to generate a visual response) are performed in parallel. If the intent recognition process determines that the question "What time is it?" is unrelated to visual processing, it immediately terminates the image review and visual reasoning processes and controls the text processing link to output a text response (i.e., the second response) based on the user's query "What time is it?" and broadcasts it via voice. In this way, the server can promptly terminate the image review and visual reasoning processes based on the intent recognition result, thereby reducing the consumption of computing resources and conserving computing resources.

[0219] Figure 7A This is a system architecture diagram illustrating the query processing method based on a large model, as described in this disclosure. Please refer to... Figure 7AIn some examples, a system applying the large-model-based query processing method of this disclosure may include both an edge and a cloud side (edge ​​side and cloud server side). The edge side is an electronic device deployed near the user, such as a smart speaker, and the cloud server side is a server capable of network communication with the edge side. The cloud server side may include an input module, an agent control center, a visual processing link, a text processing link, and other modules. The input module receives information sent by the edge side, such as query voice or images collected by the edge side, and transmits the received information to the agent control center. The agent control center forwards the information, the visual processing link processes image requests (i.e., visually related queries), and the text processing link processes non-image requests (i.e., queries unrelated to visuals).

[0220] Figure 7A In the example shown, the agent controller is the upstream module of both the visual processing link and the text processing link. When the intent recognition process in the visual processing link generates a transfer signal, the visual processing link transmits the transfer signal to the agent controller, which then determines the text response information to be output to the text processing link based on the transfer signal.

[0221] Figure 7B A timing diagram of a query processing method based on a large model provided in an embodiment of this disclosure. Figure 7B The timing diagram shown is based on Figure 7A The system architecture shown is the foundation. Please refer to... Figure 7B The time sequence of query processing methods based on large models includes: The user inputs a query into the smart speaker; The smart speaker sends an Automatic Speech Recognition (ASR) request to the agent control center, and the ASR request includes a query voice. The agent central control performs voice recognition on the query voice, obtains the user query text, and sends the user query text to the vision processing link and the text processing link respectively. The agent central control sends a photo upload command to the smart speaker; The smart speaker captures images based on the received photo upload command and uploads the captured images to the agent control center; The agent central control transmits the received images to the vision processing link; The visual processing chain processes user query text and images to generate visual responses, returns the visual responses to the agent control center, and generates access control decisions based on access control policies to indicate the output content. The text processing chain processes the user's query text, generates a text response, and returns the text response to the agent control center; The agent centrally caches visual and text responses. Upon receiving the access control decision from the visual processing link, it determines the final response based on the access control decision and returns the final response to the smart speaker. The final response can be one of the following: a visual response, a text response, or a preset prompt message. For specific decision-making methods, please refer to the aforementioned explanation of access control, which will not be repeated here.

[0222] Further reference Figure 8 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a query processing device based on a large model, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0223] like Figure 8 As shown, the query processing device 800 based on a large model in this embodiment may include: a receiving unit 801, a parallel processing unit 802, a termination unit 803, and an output unit 804. The receiving unit 801 is configured to receive user query text sent by the target device when the camera preview window of the target device is open, and to acquire suspected images collected during the process of receiving the user query text. The parallel processing unit 802 is configured to control the parallel execution of an intent recognition process, an image review process, and a visual reasoning process within the visual processing link of the large model. The intent recognition process is used to identify whether the user query text is visually relevant; the image review process is used to review suspected images to obtain image review results; and the visual reasoning process is used to generate and cache first answer information based on the user query text and suspected images. The termination unit 803 is configured to terminate the image review process and the visual reasoning process in response to the identification that the user query text is visually irrelevant. The output unit 804 is configured to control the text processing link of the large model to output second answer information based on the user query text. The text processing link and the visual processing link operate in parallel.

[0224] In this embodiment, the specific processing of the receiving unit 801, parallel processing unit 802, termination unit 803, and output unit 804 in the large model-based query processing device 800, and the resulting technical effects, can be found in reference to [reference needed]. Figure 2 The relevant descriptions of steps 201-204 in the corresponding embodiments will not be repeated here.

[0225] In some optional implementations of this embodiment, the query processing device 800 based on a large model may further include a closing unit. The closing unit is configured to: in response to not receiving user query text from the target device within a preset time period, send a preview window closing notification to the target device, so that the target device closes the camera preview window according to the preview window closing notification; or, be configured to: in response to not receiving user query text from the target device within a preset time period, send a preview window closing request to the target device, and in response to receiving a response information indicating closure corresponding to the preview window closing request, send a preview window closing notification to the target device.

[0226] In some optional implementations of this embodiment, the intent recognition process is further used to identify whether the user's query text contains an intent to exit the camera preview window. The query processing device 800 based on the large model may also include an exit unit. The exit unit is configured to: in response to receiving the user's query text within a preset time period and identifying through the intent recognition process that the user's query text contains an intent to exit the camera preview window, send a preview window closing notification to the target device, so that the target device closes the camera preview window according to the preview window closing notification.

[0227] In some optional implementations of this embodiment, the query processing apparatus 800 based on a large model may further include a visual answer output unit. The visual answer output closing unit is configured to output cached first answer information in response to recognizing that the user query text is visually relevant and that the image review result indicates that the review has been passed.

[0228] In some optional implementations of this embodiment, the query processing apparatus 800 based on a large model may further include a clearing unit. The clearing unit is configured to clear cached first and second answer information in response to recognizing that the user query text is visually relevant and the image review result indicates that the review has failed.

[0229] In some optional implementations of this embodiment, the receiving unit 810 is further configured to: determine the candidate images captured by the camera of the target device during the suspected period from the start of receiving the initial data packet constituting the user query text to receiving the end data packet of the user query text as suspected images; and receive the suspected images sent by the target device.

[0230] In some optional implementations of this embodiment, the receiving unit 810 is further configured to: in response to the fact that the candidate images captured by the camera of the target device during the suspected period are at least three images arranged in chronological order, at least one candidate image located at the beginning, middle and end of the suspected period is determined as a suspected image; wherein, the multiple suspected images are arranged in chronological order.

[0231] In some optional implementations of this embodiment, the parallel processing unit 820 is further configured to: acquire historical context related to the user's query text; and generate first answer information based on the user's query text, historical context, and suspected image.

[0232] In some optional implementations of this embodiment, the parallel processing unit 820 is further configured to: in response to the user query text being converted from the user query speech using speech-to-text technology, extract historical question and answer data related to the user query speech from the historical question and answer database of the corresponding user stored in the target device; and use the historical question and answer data as historical context related to the user query text.

[0233] In some optional implementations of this embodiment, the query processing apparatus 800 based on the large model may further include a verification unit. The verification unit is configured to: before the text processing link controlling the large model outputs second answer information based on the user query text, send a recognition result indicating that the user query text identified by the intent recognition process is visually irrelevant to the user; control the access control module to verify whether the user query text is visually irrelevant to the user; and after the access control module obtains a correction response indicating that the user query text is visually relevant, record an error indicating that the intent recognition process has an incorrect recognition result.

[0234] This embodiment exists as a device embodiment corresponding to the above method embodiment. The query processing device based on a large model provided in this embodiment can terminate the image review process and visual reasoning process in a timely manner according to the intent recognition result, thereby reducing the occupation of computing resources and saving computing resources.

[0235] According to embodiments of this disclosure, this disclosure also provides an electronic device, the electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to implement the large-model-based query processing method described in any of the above embodiments.

[0236] According to embodiments of this disclosure, this disclosure also provides a readable storage medium storing computer instructions that enable a computer to implement the large-model-based query processing method described in any of the above embodiments when executed.

[0237] According to embodiments of this disclosure, this disclosure also provides a computer program product that, when executed by a processor, can implement the query processing method based on a large model as described in any of the above embodiments.

[0238] Figure 9A schematic block diagram of an example electronic device 900 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0239] like Figure 9 As shown, the electronic device 900 includes a computing unit 901, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 902 or a computer program loaded into a random access memory (RAM) 903 from a storage unit 908. The RAM 903 can also store various programs and data for use by the electronic device 900. The computing unit 901, ROM 902, and RAM 903 are interconnected via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0240] Multiple components in electronic device 900 are connected to I / O interface 905, including: input unit 906, such as keyboard, mouse, etc.; output unit 907, such as various types of displays, speakers, etc.; storage unit 908, such as disk, optical disk, etc.; and communication unit 909, such as network card, modem, wireless transceiver, etc. Communication unit 909 allows electronic device 900 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0241] The computing unit 901 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 901 performs the various methods and processes described above, such as the large model-based query processing method. For example, in some embodiments, the large model-based query processing method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 908. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by the computing unit 901, one or more steps of the large model-based query processing method described above can be performed. Alternatively, in other embodiments, computing unit 901 may be configured to perform a large-model-based query processing method by any other suitable means (e.g., by means of firmware).

[0242] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0243] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0244] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0245] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0246] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0247] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and Virtual Private Server (VPS) services, such as high management difficulty and weak business scalability.

[0248] According to the technical solution of this disclosure, the image review process and visual reasoning process can be terminated in a timely manner based on the intent recognition result, thereby reducing the occupation of computing resources and saving computing resources.

[0249] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0250] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A query processing method based on a large model, the method comprising: With the camera preview window of the target device open, receive the user query text sent by the target device, and acquire the suspected image collected during the process of receiving the user query text; The intent recognition process, image review process, and visual reasoning process are run in parallel under the visual processing link of the large model; wherein, the intent recognition process is used to identify whether the user query text is visually relevant, the image review process is used to review the suspected image to obtain the image review result, and the visual reasoning process is used to generate and cache the first answer information based on the user query text and the suspected image. In response to the recognition that the user query text is visually irrelevant, the operation of the image review process and the visual reasoning process is terminated; The text processing link controlling the large model outputs a second answer based on the user query text; wherein the text processing link and the visual processing link run in parallel.

2. The method according to claim 1, further comprising: In response to not receiving a user query text from the target device within a preset time period, a preview window closing notification is sent to the target device so that the target device closes the camera preview window according to the preview window closing notification; or, If no user query text is received from the target device within a preset time period, a preview window closing request is sent to the target device, and if a response message indicating that the preview window should be closed is received corresponding to the preview window closing request, a preview window closing notification is sent to the target device.

3. The method according to claim 2, wherein, The intent recognition process is further used to identify whether the user's query text indicates an intent to exit the camera preview window, and the method further includes: In response to receiving the user query text within the preset time period and recognizing through the intent recognition process that the user query text contains an intent to exit the camera preview window, a preview window closing notification is sent to the target device so that the target device closes the camera preview window according to the preview window closing notification.

4. The method according to claim 1, further comprising: In response to the recognition that the user query text is visually relevant and the image review result indicates that the review has been passed, the cached first answer information is output; or, In response to the recognition that the user query text is visually relevant and the image review result indicates that the review has failed, the cached first and second answer information is cleared.

5. The method according to claim 1, wherein, The acquisition of the suspected image collected during the process of receiving the user's query text includes: The candidate images captured by the camera of the target device during the suspected period from the start of receiving the initial data packet constituting the user query text to the receipt of the end data packet of the user query text are determined as the suspected images; Receive the suspected image sent by the target device.

6. The method according to claim 5, wherein, The step of determining the candidate images captured by the camera of the target device during the period from the start of receiving the initial data packet constituting the user query text to the receipt of the final data packet constituting the user query text as the suspected images includes: In response to the fact that at least three candidate images captured by the camera of the target device during the suspected period are arranged in chronological order, at least one candidate image located at the beginning, middle and end of the suspected period is determined as the suspected image; wherein, the multiple suspected images are arranged in chronological order.

7. The method according to claim 1, wherein, The process of generating the first answer information based on the user query text and the suspected image includes: Obtain the historical context related to the user's query text; The first answer information is generated based on the user query text, the historical context, and the suspected image.

8. The method according to claim 7, wherein, The step of obtaining the historical context related to the user's query text includes: In response to the fact that the user query text is obtained by converting the user query speech through speech-to-text technology, historical question and answer data related to the user query speech is extracted from the historical voice question and answer database of the corresponding user stored in the target device; The historical question-and-answer corpus is used as the historical context related to the user's query text.

9. The method according to any one of claims 1-8, wherein, Before the text processing chain controlling the large model outputs a second answer based on the user query text, the method further includes: Send the recognition result of the user query text, which is visually irrelevant, identified by the intent recognition process to the access control management module in the text processing link; The access control management module verifies whether the user query text is visually unrelated, and after the access control management module verifies that the user query text is visually related, it records an error for the intent recognition process if the recognition result is incorrect.

10. A query processing apparatus based on a large model, comprising: The receiving unit is configured to receive user query text sent by the target device when the camera preview window of the target device is open, and to acquire suspected images collected during the process of receiving the user query text; The parallel processing unit is configured to run the intent recognition process, image review process, and visual reasoning process in parallel under the visual processing link of the large model; wherein, the intent recognition process is used to identify whether the user query text is visually relevant, the image review process is used to review the suspected image to obtain the image review result, and the visual reasoning process is used to generate and cache the first answer information based on the user query text and the suspected image. The termination unit is configured to terminate the image review process and the visual reasoning process in response to the recognition that the user query text is visually irrelevant. The output unit is configured to control the text processing link of the large model to output a second answer based on the user query text; wherein the text processing link operates in parallel with the visual processing link.

11. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the query processing method based on a large model as described in any one of claims 1-9.

12. A non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the query processing method based on a large model according to any one of claims 1-9.

13. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the query processing method based on a large model according to any one of claims 1-9.

Images