Model inference method and apparatus, electronic device, computer-readable storage medium, and computer program product

Abstract

The application provides a model reasoning method and device, electronic equipment, computer readable storage medium and computer program product; the method comprises the following steps: determining a target precision type corresponding to a model reasoning task; based on the target precision type, adjusting the length of an initial visual character sequence obtained by visual coding on an input image to determine a visual character sequence; performing model reasoning based on the visual character sequence through a large language model, and outputting a reasoning result. Through the application, the reasoning efficiency and accuracy of the model can be improved.

Description

Technical Field

[0001] This application relates to artificial intelligence technology, and more particularly to a model reasoning method, apparatus, electronic device, computer-readable storage medium, and computer program product. Background Technology

[0002] Current Large Vision-Language Models (LVLMs) encode features from image and text information before feeding them into the language model for multimodal perception and recognition. However, to improve the perception of small targets, the image input is often increased, resulting in a sharp increase in visual character unit (token) data. This leads to a significant increase in the computational cost of the Large Language Model (LLM) within the LVLM. Furthermore, excessive useless visual token data can cause model performance degradation, thereby reducing the inference efficiency and accuracy of the Large Vision-Language Model. Summary of the Invention

[0003] This application provides a model reasoning method, apparatus, electronic device, computer-readable storage medium, and computer program product that can improve the reasoning efficiency and accuracy of models.

[0004] The technical solution of this application embodiment is implemented as follows:

[0005] This application provides a model inference method, the method comprising:

[0006] Determine the target accuracy type corresponding to the model inference task;

[0007] Based on the target accuracy type, the length of the initial visual character sequence obtained by visual encoding of the input image is adjusted to determine the visual character sequence.

[0008] Using a large language model, inference is performed based on the visual character sequence, and the inference results are output.

[0009] This application provides a model inference apparatus, the apparatus comprising:

[0010] The determination module is used to determine the target accuracy type corresponding to the model inference task;

[0011] The adjustment module is used to adjust the length of the initial visual character sequence obtained by visual encoding of the input image based on the target precision type, and to determine the visual character sequence.

[0012] The reasoning module is used to perform model reasoning based on the visual character sequence using a large language model and output the reasoning results.

[0013] Optionally, the determining module is further configured to determine the target accuracy type based on the speech domain of the model inference task when the model inference task is an active perception task; and to determine the target accuracy type based on the query type of the model inference task when the model inference task is a passive perception task.

[0014] Optionally, the adjustment module is further configured to determine the target visual character sequence length based on the target precision type; and adjust the length of the initial visual character sequence obtained by visual encoding of the input image according to the target visual character sequence length to determine the visual character sequence.

[0015] Optionally, the adjustment module is further configured to adjust the length of the initial visual character sequence according to the target accuracy type and the execution frequency of the model inference task to determine a visual character sequence of a first length; and if the initial inference result obtained by performing model inference based on the visual character sequence of the first length contains the target event, adjust the length of the initial visual character sequence to determine a visual character sequence of a second length; the second length is greater than the first length.

[0016] Optionally, the reasoning module is further configured to, after determining a visual character sequence of a first length, perform model reasoning based on the visual character sequence of the first length using the large language model to obtain an initial reasoning result; and output the initial reasoning result if the initial reasoning result does not contain the target event.

[0017] Optionally, the reasoning module is further configured to, after determining the visual character sequence of the second length, perform model reasoning based on the visual character sequence of the second length through the large language model, and output the reasoning result.

[0018] Optionally, the inference module is further configured to determine the attribute identifier corresponding to the task attribute based on the task attribute of the model inference task; determine the enhanced visual character sequence by combining the visual character sequence with the attribute identifier; and perform model inference based on the enhanced visual character sequence through the large language model to output the inference result.

[0019] Optionally, the adjustment module is further configured to adjust the length of the initial visual character sequence based on the target accuracy type using a character sequence adjustment model to determine the visual character sequence; wherein the character sequence adjustment model is obtained by training the initial character sequence adjustment model; the training process includes: randomly adjusting the length of the sample visual character sequence using the initial character sequence adjustment model, and adjusting the model parameters of the initial character sequence adjustment model based on the model inference results of the sample visual character sequence after random length adjustment, until the target training conditions are met, thereby obtaining the character sequence adjustment model.

[0020] This application provides an electronic device, the electronic device comprising:

[0021] Memory is used to store executable instructions for a computer;

[0022] The processor, when executing computer-executable instructions stored in the memory, implements the model reasoning method provided in the embodiments of this application.

[0023] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the model reasoning method provided in this application when executed by a processor.

[0024] This application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, they implement the model reasoning method provided in this application.

[0025] The embodiments of this application have the following beneficial effects:

[0026] The input image is visually encoded to obtain an initial visual character sequence. The target precision type corresponding to the model's inference task is determined. Based on the target precision type, the length of the initial visual character sequence is adjusted to determine the final visual character sequence. This achieves adaptive adjustment of the length of the visually encoded visual character sequence according to the perceptual precision of the model's inference task, resulting in a visual character sequence of appropriate length that is input into the large language model for inference, yielding the inference result. This reduces the impact of redundant features on model performance, decreases the amount of token data processed by the model, and improves the model's inference efficiency and accuracy. Attached Figure Description

[0027] Figure 1 This is a schematic diagram of an optional architecture for a large visual language model;

[0028] Figure 2 This is a schematic diagram of an optional process for the model reasoning method provided in the embodiments of this application;

[0029] Figure 3 This is a schematic diagram of an optional process of the model reasoning method provided in the embodiments of this application;

[0030] Figure 4 This is a schematic diagram of an optional process of the model reasoning method provided in the embodiments of this application;

[0031] Figure 5 This is a schematic diagram of an optional process of the model reasoning method provided in the embodiments of this application;

[0032] Figure 6 This is a schematic diagram of an optional process of the model reasoning method provided in the embodiments of this application;

[0033] Figure 7 This is a schematic diagram of an optional process of the model reasoning method provided in the embodiments of this application;

[0034] Figure 8 This is a schematic diagram of an optional structure of the model inference device provided in the embodiments of this application;

[0035] Figure 9 This is a schematic diagram of an optional structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0037] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0038] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0039] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0040] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0041] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant national laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0042] Current Large Vision-Language Models (LVLMs) encode features from image and text information before feeding them into the language model for multimodal perception and recognition. However, to improve the perception of small targets, image input is often increased, leading to a sharp increase in visual token data. This causes a significant increase in the computational cost of the Large Language Model (LLM) within the LVLM. Furthermore, excessive useless visual token features can negatively impact model learning, thereby reducing the inference efficiency and accuracy of the Large Vision-Language Model.

[0043] Typically, the architecture of the LVLM model is as follows: Figure 1As shown, the model includes a Visual Transformer (ViT), a projector, and an LLM (Laser-Based Model). The cockpit image (or external image) is visually encoded by the visual encoder to obtain visual encoded features. These features are then mapped to a text feature space by the projector, resulting in a sequence of visual tokens. Since the projector can only convert unit-length visual encoded features into fixed-length visual tokens, the length of the visual token sequence corresponds to the length of the visual encoded features; the longer the visual encoded features, the longer the visual token sequence. The LLM model in the LVLM model infers from the input visual tokens and text tokens to obtain the inference result. These text tokens can be obtained by tokenizing or lexicalizing the user's command text, or by tokenizing or lexicalizing the pre-defined text corresponding to the model task. For example, the model task could be to recommend music to passengers in the cabin. The preset text for this task could be "How is the speaker feeling?" By segmenting or lexicalizing this preset text, corresponding textual tokens are obtained. An LLM model then combines these textual tokens with visual tokens corresponding to the cabin images to perform inference, yielding a result. For instance, the inference result could be "The passenger is speaking; she is happy." Based on this inference result, music matching the speaker's mood can be recommended.

[0044] In the above model processing, taking the ViT model as the openAI-CLIP-224 model as an example, when the input image size is 224*224*3, the length of the visual encoding feature output by ViT is 256*1024. If the input image size becomes 336*224*3, the length of the visual encoding feature increases to 576*1024, which leads to a corresponding increase in the sequence length of visual tokens. At this time, the computational load of the LVLM model will increase significantly, the demand for hardware resources will increase significantly, and the cost of use will increase.

[0045] Furthermore, to improve the perception of small targets, image resolution is typically increased. However, increasing image resolution introduces too many invalid and repetitive image features. Since LLM models have limited expressive and perceptual capabilities, introducing too many invalid and repetitive visual features exacerbates the difficulty of model perception and classification, thus leading to a performance regression. To address this issue, current techniques typically reduce the visual token length through feature map pruning (feature dimensionality reduction in the projector layer) or intermediate layer feature fusion (merging high-resolution features into low-resolution features while maintaining the same output feature dimension). While these methods can salvage some performance loss, they still increase computational cost, thereby reducing the efficiency of model inference.

[0046] Furthermore, current LVLM models are general-purpose solutions, but different tasks have different requirements for the model's perceptual capabilities. For example, face recognition tasks require fine-grained perception, while people counting tasks require coarse-grained perception. How to balance fine-grained and coarse-grained perception capabilities and the cost of practical inference is also an important issue.

[0047] In summary, fixed-length visual character sequences currently impose significant resource consumption and inference costs on large visual language models. Furthermore, excessively long visual character sequences can cause model performance degradation, thereby reducing model inference efficiency and accuracy.

[0048] This application provides a model inference method, apparatus, electronic device, computer-readable storage medium, and computer program product that can dynamically adjust the length of visual character sequences, thereby reducing the resource consumption and inference cost of model operation and improving model inference efficiency and accuracy.

[0049] The model inference method provided in this application can be applied to electronic devices. In some embodiments, the electronic device may include a terminal or a server. For example, the terminal may include a personal portable mobile device (such as a mobile phone, tablet, laptop, etc.) or an in-vehicle terminal; the server may include an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, network services, and big data and artificial intelligence platforms. The specific selection depends on the actual situation, and this application does not limit it.

[0050] For example, the model reasoning method of this application embodiment can be applied to intelligent perception, autonomous driving, and assisted driving based on LVLM in and out of the cabin of a vehicle, or to virtual assistants or intelligent robots of LVLM-based intelligent devices, without specific limitations.

[0051] refer to Figure 2 , Figure 2 This is a schematic diagram of an optional flowchart of the model inference method provided in an embodiment of this application. For example... Figure 2 As shown, the model inference method provided in this application embodiment can be implemented by executing the process S101-S103, as follows:

[0052] S101. Determine the target accuracy type corresponding to the model inference task.

[0053] In this embodiment, the model inference task is the inference task that the visual language large model is currently processing. The visual language large model performs the inference task through environmental perception and outputs the inference result. In some embodiments, the inference task includes active perception tasks and passive perception tasks. Active perception tasks are perception tasks initiated by the user, such as when a user wakes up a smart voice interaction assistant to instruct it to make intelligent recommendations. In this case, the smart voice interaction assistant needs to perceive the user's gender, age, facial expressions, etc., to provide more personalized intelligent recommendations. Passive perception tasks are perception tasks initiated passively. Passive perception tasks require passively and continuously perceiving the surrounding environment. For example, passive perception tasks may include monitoring whether the user is wearing a seatbelt when the vehicle starts, and continuously perceiving the number of people in the cabin while the vehicle is in motion, etc. The specific selection depends on the actual situation, and this embodiment does not limit the choice.

[0054] In this embodiment of the application, different types of reasoning tasks usually have different requirements for the accuracy and speed of model reasoning. Therefore, the reasoning tasks that the visual language large model can handle can be divided in advance according to the accuracy requirements, and the accuracy type corresponding to each reasoning task can be determined. In this way, the corresponding accuracy type can be determined as the target accuracy type according to the model reasoning task to be processed.

[0055] In some embodiments, when the model inference task is an active perception task, the target accuracy type is determined based on the speech domain of the model inference task; when the model inference task is a passive perception task, the target accuracy type is determined based on the query type of the model inference task.

[0056] In the case of an active perception task, semantic intent analysis is performed on the user's speech that triggers the active perception task to determine the speech domain. Here, the speech domain represents the model's scope where the speech intent falls. For example, the speech domain can include facial expressions, age perception, and object detection. When the user's speech's linguistic representation is used for personalized recommendations, the model needs to recognize the speaker's expression and age; therefore, the speech domain for the model inference task could be facial expressions and age perception. When the user's speech's semantic representation is searching for items inside a vehicle, the model needs to perform object detection; therefore, the speech domain for the model inference task could be object detection. It is understandable that different speech domains correspond to different use cases and have different accuracy requirements for model inference. Therefore, the accuracy type corresponding to each model's scope in the large visual language model can be predetermined. This allows the target accuracy type to be determined based on the speech domain of the current active perception task.

[0057] In the case of a passive perception task, the query type of the model inference task is determined, and the accuracy type corresponding to the query type is determined as the target accuracy type. In some embodiments, the query type of the model inference task may include an event type corresponding to at least one target event. For example, at least one target event is used to characterize at least one dangerous behavior, such as not wearing a seat belt, an impending vehicle collision, etc. Accordingly, the query type may include: seat belt monitoring, vehicle collision monitoring, etc. The specific selection is made according to the actual situation, and this application embodiment does not limit it. It is understood that different query types correspond to different use cases and have different accuracy requirements for model inference. Therefore, the accuracy types corresponding to various query types that the visual language large model can process can be determined in advance. In this way, the corresponding accuracy type can be determined as the target accuracy type according to the query type of the current passive perception task.

[0058] It should be noted that the precision of the reasoning task can also be divided from other dimensions according to the needs of the actual project. For example, the reasoning task can be classified into different precision types according to the size of the perceived target of the reasoning task in the entire input image, or according to the image resolution requirements corresponding to the reasoning task, etc. The specific selection is based on the actual situation, and this application embodiment does not limit it.

[0059] S102. Based on the target precision type, adjust the length of the initial visual character sequence obtained by visual encoding of the input image to determine the visual character sequence.

[0060] In this embodiment, the visual language large model performs visual encoding on the input image using a visual encoding model (such as ViT) to obtain visual encoding features. Then, through a mapping layer in the visual language large model, the visual encoding features are mapped to a text feature space, resulting in an initial visual character sequence. This initial visual character sequence can be visual tokens obtained by the mapping layer with a fixed length. Using the method of this embodiment, based on the target precision type determined in S101 above, the length of the initial visual character sequence can be adjusted to determine the final visual character sequence.

[0061] In some embodiments, the target visual character sequence length is determined based on the target precision type; the initial visual character sequence obtained by visual encoding of the input image is length-adjusted according to the target visual character sequence length to determine the visual character sequence.

[0062] For example, when the target precision type represents a high precision type, a longer target visual character sequence length can be determined to ensure model inference accuracy. When the target precision type represents a low precision type, a shorter target visual character sequence length can be determined to improve model inference efficiency.

[0063] In some embodiments, the length of the target visual character sequence adapted to the target precision type can be determined through model learning. That is, based on the target precision type, the length of the initial visual character sequence is adjusted, and the process of determining the visual character sequence can be performed using a character sequence adjustment model. The character sequence adjustment model is implemented through a neural network, whereby it is obtained by training an initial character sequence adjustment model. The training process includes: randomly adjusting the length of the sample visual character sequence using the initial character sequence adjustment model, and adjusting the model parameters of the initial character sequence adjustment model based on the model inference results of the randomly length-adjusted sample visual character sequence, until the target training conditions are met, thus obtaining the character sequence adjustment model.

[0064] In some embodiments, the character sequence adjustment model can be implemented based on a network model with a transformer structure. It should be noted that current transformer models, such as the QT (QueryTransformer) model, can only reduce the dimensionality of visual character sequences at a fixed ratio. However, the embodiments of this application implement dynamic dimensionality reduction of visual character sequences by dynamically adjusting their length. For example, as... Figure 3As shown, during the training phase, sample visual character sequences are input into the initial character sequence adjustment model. By randomly adjusting the sliding window length, the initial character sequence adjustment model performs inference based on visual character sequences of different lengths corresponding to sliding windows of different lengths in the sample visual character sequences, obtaining the model inference results for the training phase. During training, when the length of the visual character sequence changes, the length of the corresponding feature mapping matrix also changes accordingly, allowing the initial character sequence adjustment model to learn a set of feature mapping matrices to dynamically adjust the length of the sample visual character sequences. Simultaneously, the dynamically changing sample visual character sequences act as a regularization mechanism in model training, resulting in higher training accuracy for the same character sequence length, thus improving the model's inference accuracy. Based on the model inference results of the randomly length-adjusted sample visual character sequences, the model parameters of the initial character sequence adjustment model are adjusted until the target training conditions are met, resulting in the character sequence adjustment model. Thus, in the model application phase, the visual encoding features output by the visual encoder can be mapped through a mapping layer to obtain the initial visual character sequence. Using the trained character sequence adjustment model, the length of the initial visual character sequence is adjusted according to the target accuracy type of the current inference task to determine the final visual character sequence. The visual character sequence is sent to the large language model, which combines the visual character sequence with textual tokens to perform model inference, thereby achieving multimodal perception and outputting the inference result.

[0065] S103. Using a large language model, perform model reasoning based on visual character sequences and output the reasoning results.

[0066] In this embodiment, the large language model can be an LLM model within the LVLM model. The large language model can be deployed on the same device as the aforementioned character sequence adjustment model; alternatively, the large language model can be deployed in the cloud, while the character sequence adjustment model can be deployed locally. The large language model performs multimodal model inference based on the visual character sequence and the input text feature character sequence, outputting the inference result.

[0067] For example, the text feature character sequence can be obtained by segmenting the input text instruction, and the reasoning result can include the response information corresponding to the input text instruction.

[0068] It is understood that the embodiments of this application perform visual encoding on the input image to obtain an initial visual character sequence, determine the target precision type corresponding to the model inference task, and adjust the length of the initial visual character sequence based on the target precision type to determine the final visual character sequence. This achieves adaptive adjustment of the length of the visually encoded visual character sequence according to the perceptual precision of the model inference task, resulting in a visual character sequence of appropriate length input to the large language model for inference, thus obtaining the inference result. Consequently, the impact of redundant features on model performance is reduced, the amount of token data processed by the model is reduced, and the inference efficiency and accuracy of the model are improved.

[0069] In some embodiments, the process of S102 described above can be as follows: Figure 4 As shown, it includes:

[0070] S201. Based on the target accuracy type and the execution frequency of the model inference task, adjust the length of the initial visual character sequence to determine the first length of the visual character sequence.

[0071] S202. When model reasoning is performed based on a visual character sequence of the first length and the initial reasoning result includes the target event, the length of the initial visual character sequence is adjusted to determine a visual character sequence of the second length; the second length is greater than the first length.

[0072] In this embodiment, even though different precision levels are set for different inference tasks to correspond to different token lengths, some inference tasks need to be persistent for uninterrupted detection. Setting a high precision level for these tasks would require significant computational resources and place high performance demands on them. Therefore, for some persistent inference tasks with high precision requirements, this embodiment proposes a two-step inference method: The task type is determined based on the target precision type and the execution frequency of the model inference task. If the current model inference task is determined to be a high-precision-requirement and high-frequency inference task, the initial visual character sequence is first adjusted to a first length to determine a visual character sequence of the first length. The large language model then performs model inference based on this first-length visual character sequence to obtain the initial inference result. If the initial inference result includes a target event, the initial visual character sequence is further adjusted to determine a second-length visual character sequence; the second length is greater than the first length. The second-length visual character sequence and the event parameters corresponding to the target event are then input into the large language model again. The large language model performs model inference based on the second-length visual character sequence and the event parameters corresponding to the target event to output the inference result.

[0073] The target event can be predefined according to actual needs. For example, the target event can include dangerous events, such as vehicle overloading, passengers not wearing seat belts, vehicle fire, etc., or other events, which are not limited here. The event parameter corresponding to the target event can be the danger level corresponding to the dangerous event, which is used to help the large language model perceive the dangerous event corresponding to the danger level when performing model inference based on the second-length visual character sequence.

[0074] In some embodiments, after determining the visual character sequence of the first length, the method further includes:

[0075] Using a large language model, inference is performed based on a visual character sequence of the first length to obtain an initial inference result; if the initial inference result does not contain the target event, the initial inference result is output.

[0076] In other words, if the target event is not found in a single inference, the inference result of that inference (the initial inference result) can be directly output as the final inference result. This allows a single inference task to be completed with a smaller token length, thus improving the model's inference efficiency.

[0077] For example, such as Figure 5 As shown, the task type is first determined based on the target accuracy type and the execution frequency of the model inference task. When the current inference task is determined to be a high-accuracy and high-execution-frequency inference task based on the target accuracy type and the execution frequency of the model inference task, the length of the visual character sequence can be set as the first length. The initial visual character sequence is adjusted according to the first length to obtain the visual character sequence and output to the large language model. The large language model performs the first model inference (i.e., the first coarse perception) on the visual character sequence and the input text feature character sequence. If a non-dangerous scene is perceived (i.e., the initial inference result does not include the target event), the perception result is directly output (equivalent to outputting the initial inference result). In this way, real-time monitoring can be performed for a long time with a relatively small visual character sequence length. If a dangerous scene is detected (i.e., the initial inference result includes the target event), a second verification is performed: the visual character sequence length is adjusted and increased again, that is, the first length is increased to the second length. The large language model performs inference again based on the visual character sequence of the second length, the text feature character sequence, and the event parameters of the transmitted target event to obtain the inference result, so as to determine whether the perception result of the first coarse perception is correct. Furthermore, it can also determine whether to report the target event, such as whether to prompt the user of a dangerous scene, etc.

[0078] Understandably, by performing secondary inference verification, the detection accuracy of LVLM in dangerous scenarios is improved, balancing inference speed, inference cost, and inference accuracy, thereby improving the efficiency and accuracy of model inference.

[0079] For example, combined Figure 5 The reasoning process of LVLM can be as follows: Figure 6 As shown. When the model inference task is an active perception task, the speech domain is determined based on the user's voice that triggered the active perception task. This determines the speech domain to which the user's intention falls, such as facial expression, age perception, or object detection. Based on the speech domain of the model inference task, the target accuracy type is determined. When the model inference task is a passive perception task, the query type is determined, such as seatbelt monitoring or vehicle collision monitoring. Based on the query type, the target accuracy type is determined. For example, the target accuracy type can represent high-frequency scenarios with low accuracy requirements, or low-frequency scenarios with high accuracy requirements. The visual character sequence length is set according to the target accuracy type, and the initial visual character sequence is adjusted accordingly to obtain the final visual character sequence. The large language model in LVLM is used to perform multimodal perception on the visual character sequence and text feature character sequence. When a non-dangerous scenario is perceived, the perception result is directly output. When a dangerous scenario is perceived, secondary verification is performed. By setting a longer visual character sequence length, secondary perception is performed based on the longer visual character sequence and text feature character sequence, and the final perception result is output.

[0080] In some embodiments, the process of the above-mentioned large language model performing model inference based on visual character sequences (including a first visual character sequence or a second visual character sequence) may include:

[0081] Based on the task attributes of the model inference task, the attribute representation corresponding to the task attributes is determined; combining the visual character sequence and the attribute representation, the enhanced visual character sequence is determined; through the large language model, model inference is performed based on the enhanced visual character sequence, and the inference result is output.

[0082] In this embodiment of the application, attribute identifiers can be incorporated into the visual character sequence to suit different task attributes. For example, the attribute identifier can be...<task type> A formatted symbol or token. For example: for tasks related to personnel attributes.<task type> Can be set to<person attr> (person attr represents user attributes such as age and gender); inference tasks targeting a specific object.<task type> Can be set to<person box> (Aware of target object attributes based on user-input target bounding boxes); for hazardous events.<task type> Can be set to<Dangerous events> Wait, different attribute identifiers can be set based on different task attributes.

[0083] For example, a task attribute can correspond to one or more specific reasoning tasks, such as Figure 7 As shown, tasks 1 to 5 correspond to a task attribute, which can be represented as an attribute identifier.<task type> ,Will<task type> By combining the visual character sequence obtained from image conversion, an enhanced visual character sequence is determined. Figure 7 middle,<im_start> To enhance the delimiters in the visual character sequence, which are used to separate attribute representations from the visual character sequence; through a large language model, model inference is performed based on the enhanced visual character sequence and the text feature character sequence obtained from text segmentation, and the inference result is output.

[0084] It is understandable that by combining attribute identifiers with visual character sequences to obtain enhanced visual character sequences, task attributes can be passed to large language models to help them perform model inference, thereby improving the efficiency of model inference.

[0085] This application provides a model inference device, such as... Figure 8 As shown, the model inference device 1 may include:

[0086] Module 11 is used to determine the target accuracy type corresponding to the model inference task;

[0087] Adjustment module 12 is used to adjust the length of the initial visual character sequence obtained by visual encoding of the input image based on the target precision type, and to determine the visual character sequence;

[0088] The reasoning module 13 is used to perform model reasoning based on the visual character sequence using a large language model and output the reasoning result.

[0089] In some embodiments, optionally, the determining module 11 is further configured to determine the target accuracy type based on the speech domain of the model inference task when the model inference task is an active perception task; and to determine the target accuracy type based on the query type of the model inference task when the model inference task is a passive perception task.

[0090] In some embodiments, the adjustment module 12 is further configured to determine the target visual character sequence length based on the target precision type; and adjust the length of the initial visual character sequence obtained by visual encoding of the input image according to the target visual character sequence length to determine the visual character sequence.

[0091] In some embodiments, the adjustment module 12 is further configured to adjust the length of the initial visual character sequence according to the target accuracy type and the execution frequency of the model inference task to determine a visual character sequence of a first length; and when the initial inference result obtained by performing model inference based on the visual character sequence of the first length contains the target event, adjust the length of the initial visual character sequence to determine a visual character sequence of a second length; the second length is greater than the first length.

[0092] In some embodiments, the inference module 13 is further configured to, after determining a visual character sequence of a first length, perform model inference based on the visual character sequence of the first length using the large language model to obtain an initial inference result; and output the initial inference result if the initial inference result does not contain the target event.

[0093] In some embodiments, the reasoning module 13 is further configured to, after determining the visual character sequence of the second length, perform model reasoning based on the visual character sequence of the second length through the large language model, and output the reasoning result.

[0094] In some embodiments, the inference module 13 is further configured to determine the attribute identifier corresponding to the task attribute based on the task attribute of the model inference task; combine the visual character sequence with the attribute identifier to determine the enhanced visual character sequence; and perform model inference based on the enhanced visual character sequence through the large language model to output the inference result.

[0095] In some embodiments, the adjustment module 12 is further configured to adjust the length of the initial visual character sequence based on the target accuracy type using a character sequence adjustment model to determine the visual character sequence; wherein the character sequence adjustment model is obtained by training the initial character sequence adjustment model; the training process includes: randomly adjusting the length of the sample visual character sequence using the initial character sequence adjustment model, and adjusting the model parameters of the initial character sequence adjustment model based on the model inference results of the sample visual character sequence after random length adjustment, until the target training conditions are met, thereby obtaining the character sequence adjustment model.

[0096] It should be noted that the description of the above device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0097] This application provides an electronic device, such as... Figure 9As shown, the electronic device 3 may include: a memory 32 and a processor 33. The memory 32 and the processor 33 are connected via a communication bus 34; the memory 32 is used to store executable instructions; the processor 33 is used to implement the model reasoning method provided in this application embodiment when executing the executable instructions stored in the memory 32.

[0098] This application provides a computer-readable storage medium storing executable instructions, wherein the executable instructions are stored and, when executed by a processor, will cause the processor to execute any of the model reasoning methods provided in this application.

[0099] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0100] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0101] As an example, executable instructions may, but do not necessarily, correspond to files in the file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0102] As an example, executable instructions can be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0103] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A model reasoning method, characterized in that, The method includes: Determine the target accuracy type corresponding to the model inference task; Based on the target accuracy type, the length of the initial visual character sequence obtained by visual encoding of the input image is adjusted to determine the visual character sequence. Using a large language model, inference is performed based on the visual character sequence, and the inference results are output.

2. The method according to claim 1, characterized in that, The determination of the target accuracy type corresponding to the model inference task includes: When the model inference task is an active perception task, the target accuracy type is determined based on the speech domain of the model inference task. When the model inference task is a passive perception task, the target accuracy type is determined according to the query type of the model inference task.

3. The method according to claim 1 or 2, characterized in that, The step of adjusting the length of the initial visual character sequence obtained by visual encoding the input image based on the target precision type to determine the visual character sequence includes: Based on the target precision type, determine the length of the target visual character sequence; Based on the length of the target visual character sequence, the initial visual character sequence obtained by visual encoding of the input image is lengthened to determine the visual character sequence.

4. The method according to claim 1 or 2, characterized in that, The step of adjusting the length of the initial visual character sequence obtained by visual encoding the input image based on the target precision type to determine the visual character sequence includes: Based on the target accuracy type and the execution frequency of the model inference task, the length of the initial visual character sequence is adjusted to determine a visual character sequence of the first length; If the initial inference result obtained by performing model inference based on the visual character sequence of the first length contains the target event, the length of the initial visual character sequence is adjusted to determine a visual character sequence of the second length; the second length is greater than the first length.

5. The method according to claim 4, characterized in that, After determining the visual character sequence of the first length, the method further includes: Using the large language model, model reasoning is performed based on the visual character sequence of the first length to obtain an initial reasoning result; If the initial inference result does not contain the target event, output the initial inference result.

6. The method according to claim 5, characterized in that, After determining the visual character sequence of the second length, the method further includes: Using the large language model, model inference is performed based on the visual character sequence of the second length, and the inference result is output.

7. The method according to any one of claims 1, 2, 5, or 6, characterized in that, The process involves using a large language model to perform model inference based on the visual character sequence, and outputting the inference result, including: Based on the task attributes of the model inference task, determine the attribute identifier corresponding to the task attributes; By combining the visual character sequence with the attribute identifier, an enhanced visual character sequence is determined; Using the large language model, model inference is performed based on the enhanced visual character sequence, and the inference result is output.

8. The method according to any one of claims 1, 2, 5, or 6, characterized in that, The step of adjusting the length of the initial visual character sequence obtained by visual encoding the input image based on the target precision type to determine the visual character sequence includes: By adjusting the character sequence model based on the target accuracy type, the length of the initial visual character sequence is adjusted to determine the visual character sequence; The character sequence adjustment model is obtained by training an initial character sequence adjustment model; the training process includes: The initial character sequence adjustment model is used to randomly adjust the length of the sample visual character sequence. Based on the model inference results of the sample visual character sequence after random length adjustment, the model parameters of the initial character sequence adjustment model are adjusted until the target training conditions are met, thus obtaining the character sequence adjustment model.

9. A model reasoning device, characterized in that, The device includes: The determination module is used to determine the target accuracy type corresponding to the model inference task; The adjustment module is used to adjust the length of the initial visual character sequence obtained by visual encoding of the input image based on the target precision type, and to determine the visual character sequence. The reasoning module is used to perform model reasoning based on the visual character sequence using a large language model and output the reasoning results.

10. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions for a computer; A processor, when executing computer-executable instructions stored in the memory, implements the method according to any one of claims 1 to 8.

11. A computer-readable storage medium storing computer-executable instructions, characterized in that, When the computer-executable instructions or computer program are executed by a processor, they implement the method described in any one of claims 1 to 8.

12. A computer program product comprising computer-executable instructions, characterized in that, When the computer-executable instructions or computer program are executed by a processor, they implement the method according to any one of claims 1 to 8.

Images