Object class recognition method, display method and device based on multi-modal large model

Abstract

This application provides an object category recognition method, display method, and apparatus based on a multimodal large model, relating to the field of artificial intelligence technology. The object category recognition method based on a multimodal large model includes: acquiring a first feature image and first text information of a first object; the first feature image includes an object attribute image and an environment image of the first object; analyzing the first feature image and the first text information using a pre-trained multimodal large model to obtain a first object category prediction result for the first object; identifying target elements in the first feature image to obtain a first identification result for the target elements; and determining the target object category of the first object based on the first object category prediction result and the first identification result. This application can improve the accuracy of object category recognition and provide visual evidence for the object category recognition results, thereby enhancing the credibility of the object category recognition results.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, display method and apparatus for object category recognition based on a multimodal large model. Background Technology

[0002] With the development of big data and artificial intelligence technologies, building a standardized and intelligent object classification system has become an important path to overcome the aforementioned technical bottlenecks. In data application scenarios, object categories can be service categories of data. Service categories refer to standardized and replicable service forms formed by integrating various service elements (such as product structure, service location, service scale, service method, and service model) for specific target groups, targeting the needs of specific objects (such as merchants). Taking the food delivery industry as an example, the types of catering merchants accessed by food delivery platforms are becoming increasingly complex. There are traditional dine-in restaurants expanding their online services, as well as a large influx of pure takeout kitchens, shared stalls, and family workshops. Platforms need to accurately identify their true service forms. In related technologies, object classification often relies on manual filling and review, which is inefficient and easily circumvented. Therefore, a solution that can accurately identify object categories is needed. Summary of the Invention

[0003] This application provides a method, display method, and apparatus for object category recognition based on a multimodal large model, in order to alleviate or solve one or more technical problems existing in the prior art.

[0004] In a first aspect, embodiments of this application provide a method for object category recognition based on a multimodal large model, including:

[0005] Acquire a first feature image and first text information of a first object; the first feature image includes an object attribute image representing object attribute information of the first object, and an environment image representing service environment information of the first object; the first text information includes text description information of the first object, and a task instruction for classifying the object category of the first object;

[0006] The first feature image and the first text information are analyzed by a pre-trained multimodal large model to obtain a first object category prediction result for the first object; the target elements in the first feature image are identified to obtain a first identification result for the target elements; the first identification result is used to indicate whether the first feature image contains the target elements; the target object category of the first object is determined based on the first object category prediction result and the first identification result.

[0007] Optionally, the first object category prediction result includes the confidence level of the target object category as each candidate object category;

[0008] Determining the target object category of the first object based on the first object category prediction result and the first identification result includes:

[0009] Based on the first object category prediction result and the first recognition result, determine whether the preset object category classification conditions are met; the object category classification conditions include at least one of the following: the first feature image contains the target element, the number of target elements contained in the first feature image is greater than or equal to a preset number threshold, and the highest confidence level among the confidence levels of each candidate object category is lower than or equal to a preset confidence threshold.

[0010] If the object category classification conditions are met, the target object category is determined based on the first identification result;

[0011] If the object category classification conditions are not met, the first object category prediction result is adjusted based on the first identification result, and the target object category is determined based on the adjusted first object category prediction result.

[0012] Optionally, the target element includes a first element corresponding to a first object category and a second element corresponding to a second object category;

[0013] Determining the target object category based on the first identification result includes:

[0014] If the first recognition result indicates that the first feature image contains the first element, then the target object category is determined to be the first object category;

[0015] If the first recognition result indicates that the first feature image does not contain the second element, the target object category is determined to be the second object category.

[0016] Optionally, adjusting the first object category prediction result based on the first identification result and determining the target object category based on the adjusted first object category prediction result includes:

[0017] Based on the first identification result, the preferred object category corresponding to the first object is determined from a plurality of candidate object categories; the preferred object category is used to indicate that the probability of the target object category being the corresponding candidate object category is the highest.

[0018] Increase the confidence level of the first object category prediction result corresponding to the preferred object category by a preset amount to obtain the adjusted confidence level;

[0019] Based on the adjusted confidence level, the candidate object category with the highest confidence level is determined as the target object category.

[0020] Optionally, before acquiring the first feature image and first text information of the first object, the method further includes:

[0021] Acquire sample data; the sample data includes sample feature images, sample text information, and tag information of sample objects; the tag information includes element information of the target elements included in the sample feature images and object category identifiers of the sample object categories; the sample feature images include sample object attribute images representing the object attribute information of the sample objects, and sample environment images representing the service environment information of the sample objects; the sample text information includes text description information of the sample objects, and task instructions for classifying the object categories of the sample objects;

[0022] The sample feature image and sample text information are analyzed by a first model to obtain a second object category prediction result for the sample object; the target elements in the sample feature image are identified to obtain a second identification result for the target elements; and the predicted object category of the sample object is determined based on the second object category prediction result and the second identification result.

[0023] Based on the predicted object category, the second identification result, and the label information, the first model is iteratively trained to obtain the multimodal large model.

[0024] Optionally, the step of iteratively training the first model based on the predicted object category, the second identification result, and the label information to obtain the multimodal large model includes:

[0025] Based on the predicted object category and the object category identifier, a first loss function of the first model is determined; the first loss function is used to represent the degree of difference between the predicted object category and the sample object category.

[0026] Based on the second identification result and the feature information, a second loss function of the first model is determined; the second loss function is used to represent the degree of difference between the second identification result and the feature information.

[0027] Based on the first loss function and the second loss function, the target loss function of the first model is determined, and the model parameters of the first model are adjusted according to the target loss function to obtain the adjusted first model;

[0028] Based on the sample data, the model parameters of the adjusted first model are adjusted in the next round until the first model meets the preset iteration termination condition, thus obtaining the multimodal large model.

[0029] Optionally, the step of analyzing the sample feature image and the sample text information using the first model to obtain the second object category prediction result of the sample object includes:

[0030] Visual feature information of the sample feature image is extracted, and the sample text information is preprocessed to obtain preprocessed sample text information;

[0031] The visual feature information and the preprocessed sample text information are fused using the cross-modal attention mechanism of the first model to obtain fused multimodal feature information.

[0032] The multimodal feature information is analyzed based on the task instructions in the sample text information to obtain the second object category prediction result.

[0033] Secondly, embodiments of this application provide a method for displaying object categories based on a multimodal large model, including:

[0034] Obtain the object category identification result of the first object; the object category identification result includes the target object category; the object category identification result is obtained according to the object category identification method based on a multimodal large model described in any of the above embodiments;

[0035] The object category identification result is displayed at the display position of the first object.

[0036] Optionally, the object category recognition result is displayed at the display location of the first object, including:

[0037] At a designated location in the object attribute image of the first object, an object category identifier corresponding to the target object category is displayed; the display location includes the designated location.

[0038] And / or,

[0039] In the display area of ​​the object attribute information of the first object, the feature information of the target elements contained in the first feature image of the first object is displayed; the display location includes the display area; the object category recognition result also includes the feature information.

[0040] Thirdly, embodiments of this application provide an object category recognition device based on a multimodal large model, including:

[0041] The first acquisition module is used to acquire a first feature image and first text information of a first object; the first feature image includes an object attribute image representing object attribute information of the first object and an environment image representing service environment information of the first object; the first text information includes text description information of the first object and a task instruction for classifying the object category of the first object.

[0042] The recognition module is used to analyze the first feature image and the first text information using a pre-trained multimodal large model to obtain a first object category prediction result for the first object; to recognize the target elements in the first feature image to obtain a first recognition result for the target elements; the first recognition result is used to indicate whether the first feature image contains the target elements; and to determine the target object category of the first object based on the first object category prediction result and the first recognition result.

[0043] Fourthly, embodiments of this application provide an object category display device based on a multimodal large model, including:

[0044] The second acquisition module is used to acquire the object category identification result of the first object; the object category identification result includes the target object category; the object category identification result is obtained according to the object category identification method based on a multimodal large model described in any of the above embodiments;

[0045] The display module is used to display the object category recognition result at the display position of the first object.

[0046] Fifthly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor implements any of the methods of embodiments of this application when executing the computer program.

[0047] Sixthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method of any one of the embodiments of this application.

[0048] In a seventh aspect, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the method of any one of the embodiments of this application.

[0049] According to the technical solution of this application embodiment, by acquiring a first feature image and first text information of a first object, the first feature image and first text information are input into a pre-trained multimodal large model to analyze the first feature image and first text information through the multimodal large model to obtain a first object category prediction result for the first object; and by identifying target elements in the first feature image to obtain a first identification result for the target elements; and then, based on the first object category prediction result and the first identification result, the target object category of the first object is determined. It can be seen that this application, through an end-to-end multimodal reasoning process, not only utilizes a multimodal large model to perform deep fusion reasoning on image visual information and text information, achieving the effect of automatically and accurately identifying object categories from multi-source heterogeneous data of images and text, avoiding misjudgments based on hard rules, but also, by combining the first identification result of the object attribute image and the environment image of the first object, since the first identification result is used to indicate whether the first feature image contains target elements, the target elements contained in the first feature image can provide reliable data support for the finally identified object category, improving the accuracy and reliability of object category identification.

[0050] The technical solution of this application can be applied to the transaction and delivery services of instant e-commerce platforms, such as Taobao Flash Sale, Taoxianda, Ele.me takeaway and retail.

[0051] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application, it can be implemented according to the contents of the specification. In order to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0052] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments according to this application and should not be construed as limiting the scope of this application.

[0053] Figure 1 A flowchart of the object category recognition method based on a multimodal large model provided in this application is shown.

[0054] Figure 2 The flowcharts of object category recognition methods based on multimodal large models provided in other embodiments of this application are shown.

[0055] Figure 3 A flowchart of the object category display method based on a multimodal large model provided in this application is shown;

[0056] Figure 4This diagram shows a block diagram of an object category recognition device based on a multimodal large model provided in an embodiment of this application;

[0057] Figure 5 This diagram illustrates a block diagram of an object category display device based on a multimodal large model provided in an embodiment of this application.

[0058] Figure 6 A block diagram of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0059] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the concept or scope of this application. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0060] To facilitate understanding of the technical solutions of the embodiments of this application, the relevant technologies of the embodiments of this application are described below. The following relevant technologies are optional solutions and can be combined with the technical solutions of the embodiments of this application in any way, and all of them fall within the protection scope of the embodiments of this application.

[0061] The following terms will be used in the following text:

[0062] Service category: refers to a standardized and replicable service form formed by integrating various service elements (such as product structure, service location, service scale, service method, service model, etc.) for a specific target group within a specific field. Taking catering merchants as an example, the service category is the actual physical form and operating model of the services provided by catering merchants to the outside world.

[0063] Multimodal large models: Large pre-trained models capable of simultaneously processing multiple input modalities such as images and text, and performing cross-modal semantic understanding and reasoning.

[0064] Prompt: A technique used during the inference phase of a large model to guide the model to focus on a specific task and output structured results through carefully designed natural language instructions.

[0065] SFT (Supervised Fine-Tuning): Based on a pre-trained large model, supervised training is performed using high-quality labeled data to adapt to specific downstream tasks and improve the model's performance and stability in the target scenario.

[0066] Storefront Image: Images uploaded by merchants, including store signs, entrance appearance, etc., used to showcase the merchant's external image and geographical location.

[0067] Environment photos: Images of the interior scenes uploaded by merchants, such as the kitchen work area, dining area, and cashier, reflecting the actual service environment and service configuration of the merchants.

[0068] This application aims to provide an object category recognition method based on a multimodal large model. It acquires a first feature image and first text information of a first object, inputs these into a pre-trained multimodal large model, and analyzes them to obtain a first object category prediction result. It also identifies target elements in the first feature image to obtain a first identification result for the target elements. Finally, based on the first object category prediction result and the first identification result, the target object category of the first object is determined. This application, through an end-to-end multimodal inference process, not only utilizes a multimodal large model for deep fusion inference of image visual information and text information, achieving automatic and accurate object category recognition from multi-source heterogeneous data of images and text, but also avoids misjudgments based on hard rules. Furthermore, by combining the first identification results of the object attribute image and the environment image of the first object, since the first identification result indicates whether the first feature image contains target elements, the target elements contained in the first feature image can provide reliable data support for the finally identified object category, improving the accuracy and reliability of object category recognition.

[0069] The object category recognition method based on a multimodal large model provided in this application can be applied to online transaction service scenarios, including online shopping and food delivery. The object can be a merchant operating a goods transaction service through an online platform. Taking food delivery as an example, the object is a merchant providing food delivery services. The merchant's object category can be a service category, which can include dine-in, food stalls, or takeaway kitchens. Dine-in refers to a fixed location with an independent storefront, a frontage open to the public, and a place for customers to dine. Food stalls typically do not have an independent storefront and share public dining space, such as fixed window-style operating units located in food courts, shopping malls, or shared dining areas in office buildings. Takeaway kitchens only support online order production, have no frontage open to the public, and are usually located in non-street-facing buildings or shared kitchens.

[0070] It should be noted that the application scenarios or examples provided in the embodiments of this application are for ease of understanding, and the embodiments of this application do not specifically limit the application of the technical solutions. In addition, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0071] The technical solution of this application and how it solves the aforementioned technical problems are described in detail below with specific embodiments. The listed specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0072] Figure 1 The flowchart of the object category recognition method based on a multimodal large model provided in this application is shown, as follows: Figure 1 As shown, the method may include steps S101 and S102.

[0073] Step S101: Obtain a first feature image and first text information of the first object; the first feature image includes an object attribute image representing the object attribute information of the first object and an environment image representing the service environment information of the first object; the first text information includes text description information of the first object and a task instruction for classifying the object category of the first object.

[0074] The object attribute information of the first object may include at least one of the following: object name, service category, service time, service address, service mode, etc. Service environment information includes item information of the physical items present within the service environment. Text description information may include at least one of the following: object attribute information, comment information, etc. Comment information can be obtained by collecting comments from any user on the first object. Text description information may include structured text content and / or unstructured text content. The task instruction is called Prompt, used to guide the multimodal large model to classify the object category of the first object.

[0075] Taking the food delivery scenario as an example, the first object is the merchant providing the delivery service. The merchant object category can include categories such as dine-in, food stalls, or delivery kitchens. The merchant object attribute image can be a storefront image, and the service environment image can be an image of the store's interior environment.

[0076] Optionally, after obtaining the first feature image, the first feature image can be standardized, for example, the image size of the first feature image can be normalized to 512*512, and deblurring and illumination equalization algorithms can be applied to improve the image quality.

[0077] Step S102: Analyze the first feature image and the first text information using a pre-trained multimodal large model to obtain the first object category prediction result of the first object; identify the target elements in the first feature image to obtain the first identification result of the target elements; the first identification result is used to indicate whether the first feature image contains target elements; determine the target object category of the first object based on the first object category prediction result and the first identification result.

[0078] The first object category prediction result includes the confidence level of the target object category as each candidate object category, or the first object category prediction result includes the candidate object category with the highest confidence level and the highest confidence level. The first identification result includes whether the first feature image contains target elements. Furthermore, if the first identification result includes "the first feature image contains target elements", the first identification result may also include the number of target elements contained in the first feature image.

[0079] The first identification result of the target element may include at least one of the following: the element category of the target element, the number of target elements in the first feature image, etc.

[0080] The multimodal large model is trained based on sample feature images, sample text information, and label information of multiple sample objects. The label information includes feature information of target elements included in the sample feature images and object category identifiers of sample object categories; the sample feature images include sample object attribute images used to represent object attribute information and sample environment images used to represent service environment information of sample objects; the sample text information includes text description information of sample objects and task instructions used to indicate the object category of the sample objects.

[0081] According to the technical solution of this application embodiment, by acquiring a first feature image and first text information of a first object, the first feature image and first text information are input into a pre-trained multimodal large model to analyze the first feature image and first text information through the multimodal large model to obtain a first object category prediction result for the first object; and by identifying target elements in the first feature image to obtain a first identification result for the target elements; and then, based on the first object category prediction result and the first identification result, the target object category of the first object is determined. It can be seen that this application, through an end-to-end multimodal reasoning process, not only utilizes a multimodal large model to perform deep fusion reasoning on image visual information and text information, achieving the effect of automatically and accurately identifying object categories from multi-source heterogeneous data of images and text, avoiding misjudgments based on hard rules, but also, by combining the first identification result of the object attribute image and the environment image of the first object, since the first identification result is used to indicate whether the first feature image contains target elements, the target elements contained in the first feature image can provide reliable data support for the finally identified object category, improving the accuracy and reliability of object category identification.

[0082] In some embodiments, the first object category prediction result includes the confidence score of the target object category as each candidate object category. The candidate object categories are a plurality of pre-configured object categories. When determining the target object category of the first object based on the first object category prediction result and the first identification result, the following steps A1 and A2 can be performed:

[0083] Step A1: Based on the first object category prediction result and the first recognition result, determine whether the preset object category classification conditions are met.

[0084] The object category classification conditions may include at least one of the following: the first feature image contains target elements, the number of target elements contained in the first feature image is greater than or equal to a preset number threshold, and the highest confidence level among the confidence levels of each candidate object category is lower than or equal to a preset confidence threshold.

[0085] Step A2: If the object category classification conditions are met, determine the target object category based on the first identification result. If the object category classification conditions are not met, adjust the first object category prediction result based on the first identification result, and determine the target object category based on the adjusted first object category prediction result.

[0086] Optionally, target elements corresponding to various candidate object categories are pre-defined. When determining the target object category based on the first recognition result, if the first recognition result includes the first feature image containing target elements, and / or the number of target elements contained in the first feature image is greater than or equal to a preset number threshold, then the target object category of the first object is the candidate object category corresponding to the target elements contained in the first feature image.

[0087] In this embodiment, when the object category classification conditions are met, it indicates that the reliability of the first identification result is higher than the reliability of the first object category prediction result. Therefore, the first identification result can override the first object category prediction result, and the first identification result can be used as the standard for determining the target object category. When the object category classification conditions are not met, it indicates that the reliability of the first identification result is not necessarily higher than the reliability of the first object category prediction result. That is, the target object category of the first object cannot be accurately determined directly based on the first identification result. In this case, comprehensively analyzing the first identification result and the first object category prediction result to determine the target object category of the first object helps to improve the accuracy and robustness of object category identification.

[0088] In some embodiments, the target element includes a first element corresponding to a first object category and a second element corresponding to a second object category.

[0089] When determining the target object category based on the first recognition result, if the object category classification conditions are met, the following steps can be performed: If the first recognition result indicates that the first feature image contains a first element, the target object category is determined to be the first object category. If the first recognition result indicates that the first feature image does not contain a second element, the target object category is determined to be the second object category.

[0090] It should be noted that the above method of determining the target object category based on the first recognition result is only a partial example. In practical applications, many other determination methods can be used. For example, if the first recognition result indicates that the first feature image contains a first element and the number of the first element is greater than or equal to a preset quantity threshold, the target object category is determined to be the first object category. As another example, if the target element also includes a third element corresponding to a third object category, if the first recognition result indicates that the first feature image only contains the third element, the target object category is determined to be the third object category.

[0091] Taking the food delivery scenario as an example, three candidate categories are pre-defined: dine-in, food stalls, and takeaway kitchens. The target elements for the "dine-in" category include dining tables and a separate storefront; the target elements for the "food stall" category include the words "XX Food City" and an open work surface; and the target elements for the "takeaway kitchen" category include printed paper stickers. The first feature image is the storefront image provided by the merchant, and the environment image is the interior environment image provided by the merchant.

[0092] The preset object category classification conditions include at least one of the following:

[0093] (1) The first feature image contains the first element "dining table", and the number of the first element "dining table" is greater than or equal to a preset number threshold, such as 2.

[0094] (2) The first feature image contains the second element "open operating table".

[0095] (3) The first feature image contains only the third element, “the content of the printed paper sticker”.

[0096] (4) The highest confidence level among the confidence levels of each candidate object category is lower than or equal to the preset confidence threshold, such as 0.6.

[0097] If the first identification result is that the store environment map includes two dining tables, it means that the above object category classification condition (1) is met. At this time, it can be determined that the merchant's target object category is the object category "dine-in" corresponding to the first element. If the first identification result is that the store environment map includes an open worktable, it means that the above object category classification condition (2) is met. At this time, it can be determined that the merchant's target object category is the object category "stall" corresponding to the second element. If the first identification result is that the storefront map only contains printed paper stickers, it means that the above object category classification condition (3) is met. At this time, it can be determined that the merchant's target object category is the object category "take-out kitchen" corresponding to the third element.

[0098] For object category classification condition (4), if the highest confidence level among the confidence levels of each candidate object category is lower than or equal to the preset confidence threshold, it indicates that the confidence level of the first object category prediction result is low. At this time, the target object category is determined based on the first identification result. Specifically, it can be determined which object category classification condition in (1) to (3) above the first identification result satisfies, and then the target object category of the merchant is determined based on the object category classification condition that is satisfied.

[0099] In some embodiments, when the object category classification conditions are not met, adjusting the first object category prediction result based on the first identification result and determining the target object category based on the adjusted first object category prediction result can be performed as follows: Steps B1, B2, and B3.

[0100] Step B1: Based on the first identification result, determine the preferred object category corresponding to the first object from multiple candidate object categories. The preferred object category is used to indicate that the target object category has the highest probability of being the corresponding candidate object category.

[0101] Step B2: Increase the confidence level of the first object category prediction result corresponding to the preferred object category by a preset amount to obtain the adjusted confidence level.

[0102] Step B3: Based on the adjusted confidence level, determine the candidate object category with the highest confidence level as the target object category.

[0103] For example, if the first recognition result is that the first feature image contains a first element, then the preferred object category corresponding to the first object can be determined as the candidate object category corresponding to the first element. If the first recognition result is that the first feature image contains a second element, then the preferred object category corresponding to the first object can be determined as the candidate object category corresponding to the second element.

[0104] The preset magnitudes for different preference categories can be the same or different. For example, when the preference category is a candidate category corresponding to the first factor, the confidence level corresponding to that candidate category is increased by 0.2; when the preference category is a candidate category corresponding to the second factor, the confidence level corresponding to that candidate category is increased by 0.3; and so on. Furthermore, when increasing the confidence level of a candidate category, the confidence level of other candidate categories that are opposed to that candidate category can be decreased.

[0105] In some embodiments, a multimodal large model is pre-trained before acquiring the first feature image and first text information of the first object. For example... Figure 2 As shown, the training process of the multi-model large model includes the following steps S201, S202, and S203:

[0106] Step S201: Obtain sample data; the sample data includes sample feature images of sample objects, sample text information, and label information; the label information includes feature information of target elements included in the sample feature images and object category identifiers of sample object categories.

[0107] The sample feature image includes a sample object attribute image representing the object attribute information of the sample object, and a sample environment image representing the service environment information of the sample object; the sample text information includes text description information of the sample object, and task instructions for classifying the object category of the sample object.

[0108] The object attribute information of the sample object may include at least one of the following: object name, service category, service time, service address, service mode, etc. The text description information of the sample object may include at least one of the following: object attribute information, comment information, etc. Comment information can be obtained by collecting comments from any user on the sample object. The text description information may include structured text content and / or unstructured text content. The task instruction is called Prompt, which is used to guide the multimodal large model to classify the sample object into object categories.

[0109] The feature information of the target feature may include at least one of the following: the feature category of the target feature, the number of the target feature in the sample feature image, etc.

[0110] Step S202: Analyze the sample feature image and sample text information using the first model to obtain the second object category prediction result of the sample object; identify the target elements in the sample feature image to obtain the second identification result of the target elements; determine the predicted object category of the sample object based on the second object category prediction result and the second identification result.

[0111] Optionally, when analyzing the sample feature image and sample text information using the first model, firstly, the visual feature information of the sample feature image is extracted, and the sample text information is preprocessed to obtain preprocessed sample text information; secondly, the cross-modal attention mechanism of the first model is used to fuse the visual feature information and the preprocessed sample text information to obtain fused multimodal feature information; thirdly, the multimodal feature information is analyzed based on the task instructions to obtain the second object category prediction result.

[0112] The second identification result of the target element may include at least one of the following: the element category of the target element, the number of target elements in the sample feature image, etc.

[0113] Step S203: Based on the predicted object category, the second recognition result, and the label information, the first model is iteratively trained to obtain a multimodal large model.

[0114] In this embodiment, the first model can be any type of large-scale visual-language model; optionally, the first model is a Qwen-VL model. The Qwen-VL model supports tasks such as text-to-image question answering, image description, and cross-modal retrieval, and possesses powerful zero-shot and few-shot inference capabilities. By performing SFT (Supervised Fine-tuning) on ​​the first model, a large-scale multimodal model can be obtained.

[0115] In some embodiments, the second object category prediction result includes the confidence score of the target object category for each candidate object category. The candidate object categories are a pre-configured plurality of object categories. When determining the predicted object category of a sample object based on the second object category prediction result and the second identification result, the following steps C1 and C2 can be performed:

[0116] Step C1: Based on the second object category prediction result and the second identification result, determine whether the preset object category classification conditions are met.

[0117] The object category classification conditions may include at least one of the following: the sample feature image contains target elements, the number of target elements in the sample feature image is greater than or equal to a preset number threshold, and the highest confidence level among the confidence levels of each candidate object category is lower than or equal to a preset confidence threshold.

[0118] Step C2: If the object category classification conditions are met, determine the predicted object category based on the second identification result. If the object category classification conditions are not met, adjust the confidence level of each candidate object category based on the second identification result, and determine the predicted object category based on the adjusted confidence level.

[0119] Optionally, target elements corresponding to various candidate object categories are pre-defined. When determining the predicted object category based on the second recognition result, if the second recognition result includes: the sample feature image contains target elements, and / or the number of target elements contained in the sample feature image is greater than or equal to a preset number threshold, then the predicted object category of the sample object is the candidate object category corresponding to the target elements contained in the sample feature image.

[0120] The execution method of step C1 is the same as that of step A1 in the above embodiment, and the execution method of step C2 is the same as that of step A2 in the above embodiment, the only difference being the processing object, which is the first object and the sample object respectively. Therefore, the detailed execution method will not be repeated.

[0121] Optionally, step S203 can be performed as the following steps D1, D2, and D3:

[0122] Step D1: Determine the first loss function of the first model based on the predicted object category and the object category identifier; the first loss function is used to represent the degree of difference between the predicted object category and the sample object category.

[0123] Step D2: Based on the second identification result and the feature information, determine the second loss function of the first model; the second loss function is used to represent the degree of difference between the second identification result and the feature information.

[0124] Step D3: Determine the target loss function of the first model based on the first loss function and the second loss function, and adjust the model parameters of the first model according to the target loss function to obtain the adjusted first model; adjust the model parameters of the adjusted first model in the next round according to the sample data until the first model meets the preset iteration termination condition to obtain the multimodal large model.

[0125] The preset iteration termination condition may include at least one of the following: the number of iterations reaches a preset threshold, or the value of the target loss function is less than or equal to a preset loss threshold.

[0126] In this embodiment, the target loss function of the first model reflects both the difference between the predicted object category and the sample object category, and the difference between the second recognition result of the target element in the sample feature image and the element information of the target element. Therefore, the target loss function encourages the model to focus on visual regions consistent with the label information, enabling the trained multimodal large model to not only accurately identify the target object category, avoiding misjudgments based solely on hard rules, but also to provide reliable data support for the final identified target object category by recognizing the target elements in the object's feature image, thus improving the accuracy and reliability of object category recognition. Furthermore, the Prompt function explicitly guides the model to output the judgment criteria, making the final object category recognition result auditable and operable, thereby enhancing the trustworthiness of object category recognition.

[0127] The following section uses a food delivery scenario as an example to explain in detail the object category recognition method provided in this application. In the food delivery scenario, the object is the merchant, and the merchant's object category is the service category, which can include categories such as dine-in, food stalls, or takeaway kitchens. The object category recognition method mainly includes the training and inference processes of a multimodal large model.

[0128] Before model training, a first model needs to be determined. In this embodiment, the first model is the Qwen-VL model, which includes a post-processing module. This module is used to perform image recognition on the feature images to identify target elements within them. During model training, sample data is first acquired and preprocessed. The sample data includes: sample storefront images, sample environment images, text descriptions of the sample merchants, prompts indicating the service categories of the sample merchants, and tag information. For each sample merchant, at least one sample storefront image and at least one sample environment image can be provided. The environment image may include a kitchen, dining area, cashier, etc. The text descriptions of the sample merchants may include at least one of the following: merchant attribute information, user reviews of the sample merchant, etc. Merchant attribute information may include at least one of the following: merchant name, service category, service time, service address, service mode, etc. After obtaining the sample data, the sample feature images (including sample gate images and sample environment images) can be standardized. For example, the image size of the sample feature images can be normalized to 512*512, and deblurring and illumination equalization algorithms can be applied to improve image quality.

[0129] In this embodiment, the sample object category of the sample merchant can be the sample service category, and the object category identifier can be the service category identifier. The label information includes the element information of the target elements included in the sample feature image and the service category identifier of the sample service category. To ensure the high quality of the sample data, the target elements of each sample feature image can be pre-annotated as key visual evidence. For example, for dine-in merchants, the environmental image of the merchant can be labeled "The environmental image contains at least 2 tables" as visual evidence that the merchant belongs to the dine-in service category.

[0130] An example prompt is as follows: "You are a local services platform review expert. Based on the following image and description, determine the true service category of this merchant. Options: A. Dine-in (with a separate store and fixed dining area) B. Stall (located in a food court / shopping mall, with a fixed window and shared dining area) C. Pure takeout kitchen (no storefront, no dine-in area, only supports online orders). Please output the options and confidence level."

[0131] Secondly, the sample data is input into the Qwen-VL model. Specifically, the sample feature images are input into the visual encoder of the Qwen-VL model, which extracts global and local visual features. Sample text information is concatenated into contextual information; for example, multiple scattered sample text messages are concatenated into a complete contextual message, "Mainly serving Sichuan cuisine, located at A08, Food Court, B1 Floor, XX Shopping Center." This contextual information is then input into the text encoder of the Qwen-VL model, which extracts textual features. Next, the cross-modal attention mechanism of the Qwen-VL model is used to fuse the visual and textual features, resulting in fused multimodal feature information. The fusion process includes establishing semantic alignment between key image regions and text keywords. For example, establishing semantic alignment between key image regions containing "dining table" and "signboard" and the text keywords "dining table" and "signboard" facilitates semantic analysis of the multimodal feature information by the model. Based on the semantic analysis results, the service category of the sample merchant is predicted, resulting in a second service category prediction. This second service category prediction can include the candidate service category with the highest confidence and the highest confidence level. For example, if the Qwen-VL model predicts the candidate service category with the highest confidence in the current iteration with a confidence of 0.7, and that candidate service category is "dine-in", then the output of the Qwen-VL model could be "dine-in + 0.7".

[0132] Subsequently, the target elements in the sample environment map are identified through the post-processing module to obtain the second identification result of the target elements. Based on the second service category prediction result and the second identification result, the predicted service category of the sample merchant is determined.

[0133] Next, based on the predicted service category and service category identifier, the first loss function of the Qwen-VL model is determined. This first loss function can be calculated using cross-entropy and represents the difference between the predicted service category and the sample service category. Based on the second identification result and the element information in the label information, the second loss function of the Qwen-VL model is determined. This second loss function can be calculated using a reward mechanism and represents the difference between the second identification result and the element information. For example, if the element information is the element category of the target element, a positive reward is given if the element category of the target element in the second identification result matches the element category in the label information; a negative reward is given if the element category of the target element in the second identification result does not match the element category in the label information. Then, the first and second loss functions are weighted and calculated to obtain the target loss function of the Qwen-VL model. The model parameters of the Qwen-VL model are then adjusted according to the target loss function to obtain a multimodal large model.

[0134] Optionally, during model training, sample data with confidence levels lower than a preset confidence threshold can be automatically entered into a manual review queue. After manual review, the review results are used for continuous model iteration, thereby establishing a closed-loop system of "automatic identification - manual review - model iteration". This system supports dynamic updates of large-scale fuzzy merchants and coverage of long-tail scenarios, and has strong engineering implementation capabilities.

[0135] It is understandable that training a multimodal large-scale model using the above methods and constructing a multi-evidence fusion mechanism of "entrance image + environment image + text," coupled with collaborative verification through visual orthogonality and semantic context, enables the multimodal large-scale model to possess both common-sense reasoning ability—understanding that "no tables and chairs" does not necessarily equate to "definitely no dine-in service"—thus avoiding misjudgments based on hard rules, while also ensuring accuracy and robustness. Secondly, since the model training relies on both image and text data, it can depend on text data when image data is blurry, and on visual evidence when text data is missing (e.g., comments are missing), achieving a complementary effect between multimodal approaches. Furthermore, by explicitly guiding the model to output judgment criteria through a Prompt function, cross-modal causal reasoning is performed to verify domain logic, making the service category identification results auditable and operable, thus enhancing the credibility of the service category identification results. Finally, because the Qwen-VL model possesses powerful zero-shot and few-shot reasoning capabilities, only a small number of samples are needed to fine-tune the Qwen-VL model to obtain a multimodal large-scale model adapted to new service categories, reducing model training costs. Furthermore, the multimodal large model trained in this way can accurately understand Chinese concepts of various service categories such as "dine-in" and "stall", reduce semantic drift, lower the proportion of low-confidence samples, and significantly improve the model's understanding accuracy and stability in Chinese contexts.

[0136] The following describes the application inference process of the multimodal large model. First, the feature images and text information of the merchants are acquired. The feature images include storefront images and environmental images, while the text information includes textual descriptions and a prompt used to classify the merchant's service category. The textual descriptions may include at least one of the following: merchant attribute information, user reviews of the merchant, etc. Merchant attribute information may include at least one of the following: merchant name, service category, service time, service address, service mode, etc. The prompt used to classify the merchant's service category is the same as the prompt used during model training and will not be repeated here.

[0137] Then, the merchant's feature image and text information are input into the trained multimodal large-scale model. The multimodal large-scale model analyzes the feature image and text information to obtain the merchant's first service category prediction result, such as "dine-in +0.8". Furthermore, the post-processing module in the multimodal large-scale model identifies the merchant's feature image to obtain the first identification result of the target element. The first identification result includes whether the feature image contains the target element. If the feature image contains the target element, the first identification result may also include the number of target elements contained in the feature image.

[0138] Then, based on the prediction results of the first service category and the first identification results, the merchant's target service category is determined.

[0139] In this embodiment, it is assumed that the target elements corresponding to the candidate service category "Dine-in" include dining tables and an independent storefront; the target elements corresponding to the candidate service category "Stall" include the words "XX Food City" and an open worktable; and the target elements corresponding to the candidate service category "Takeout Kitchen" include printed paper stickers. The preset service category classification conditions include at least one of the following: the environment image includes "dining tables," and the number of "dining tables" is greater than or equal to 2; the environment image includes an "open worktable"; the feature image does not include a storefront image; and the storefront image only includes "printed paper stickers." As long as the first identification result meets any one of these service category classification conditions, the merchant's target service category is determined based on the first identification result. If none of the service category classification conditions are met, the confidence level of each candidate service category is adjusted based on the first identification result, and the merchant's target service category is determined based on the adjusted confidence level.

[0140] The following are some examples of the initial identification results and the determination results of the target service category:

[0141] If the first identification result is: ≥2 dining tables are detected in the environment map and the storefront image shows an independent storefront, then the merchant's target service category is determined to be "dine-in", or the confidence level of "dine-in" is strengthened.

[0142] If the first identification result is: the storefront image contains the words "XX Food City" and the environment image includes an open worktable, then the merchant's target service category is determined to be "stall," or the preferred service category is determined to be "stall," meaning the probability of the target service category being "stall" is the highest. Then, the merchant's target service category is determined by combining the preferred service category and the first service category prediction results. Optionally, the confidence level of the candidate service category "stall" can be increased, for example, by 0.2, and then the merchant's target service category is determined based on the adjusted confidence level.

[0143] If the first identification result is: there is no storefront image in the feature image or the storefront image is just a printed paper sticker, and there is no dining area in any of the environmental images, then the merchant's target service category is determined to be "pure takeout kitchen".

[0144] It is understandable that by adopting the aforementioned object category recognition method, employing a three-in-one architecture of a supervised fine-tuned multimodal large model, structured prompt, and visual evidence calibration, compared to text rule matching schemes, this application directly perceives physical space from visual evidence, avoiding keyword avoidance and greatly improving the accuracy of service category recognition. Compared to single-modal image models, this application's multimodal large model integrates storefront images, environmental images, and textual information, achieving multi-view cross-validation, especially significantly improving the recognition accuracy for "dine-in" service categories, accurately distinguishing the subtle differences between "stall" and "dine-in" service categories. Furthermore, the entire service category recognition process requires no manual review, enabling automated recognition of service categories for a massive number of merchants simultaneously, greatly reducing labor costs.

[0145] Figure 3 The flowchart illustrates the object category display method based on a multimodal large model provided in this application embodiment, such as... Figure 3 As shown, the method may include steps S301 and S302.

[0146] Step S301: Obtain the object category recognition result of the first object, which includes the target object category.

[0147] The object category identification result is obtained according to the object category identification method based on multimodal large model in any of the above embodiments.

[0148] Step S302: Display the object category recognition result at the display position of the first object.

[0149] In some embodiments, displaying the object category identification result at the display location of the first object can be performed as follows:

[0150] At a specified location on the object attribute image of the first object, display the object category identifier corresponding to the target object category; the display location includes the specified location;

[0151] And / or,

[0152] In the display area of ​​the object attribute information of the first object, the feature information of the target elements contained in the first feature image of the first object is displayed; the display location includes the display area; the object category recognition result also includes feature information.

[0153] In the above embodiments, by displaying the object category identification results at the display location of the first object, for example, by displaying the object category identifier corresponding to the target object category on the merchant's storefront image, users can quickly understand the target object category of an object when browsing various objects, so that users can place orders on the platform as needed. Furthermore, by displaying the element information of the target elements contained in the first feature image of the first object in the display area of ​​the object attribute information of the first object, users can simultaneously understand the element information matching the target object category when viewing the target object category, i.e., visual evidence, thereby strengthening their trust in the target object category.

[0154] Corresponding to the application scenarios and methods provided in the embodiments of this application, the embodiments of this application also provide an object category recognition device based on a multimodal large model and an object category display device based on a multimodal large model.

[0155] Figure 4 A block diagram of an object category recognition device based on a multimodal large model provided in an embodiment of this application is shown, such as... Figure 4 As shown, the object category recognition device 40 based on a multimodal large model includes:

[0156] The first acquisition module 41 is used to acquire a first feature image and first text information of a first object; the first feature image includes an object attribute image representing object attribute information of the first object and an environment image representing service environment information of the first object; the first text information includes text description information of the first object and a task instruction for classifying the object category of the first object.

[0157] The recognition module 42 is used to analyze the first feature image and the first text information using a pre-trained multimodal large model to obtain a first object category prediction result for the first object; to recognize the target element in the first feature image to obtain a first recognition result for the target element; the first recognition result is used to indicate whether the first feature image contains the target element; and to determine the target object category of the first object based on the first object category prediction result and the first recognition result.

[0158] In some embodiments, the first object category prediction result includes the confidence level of the target object category as each candidate object category;

[0159] When the identification module 42 determines the target object category of the first object based on the first object category prediction result and the first identification result, it performs the following steps:

[0160] Based on the first object category prediction result and the first recognition result, determine whether the preset object category classification conditions are met; the object category classification conditions include at least one of the following: the first feature image contains the target element, the number of target elements contained in the first feature image is greater than or equal to a preset number threshold, and the highest confidence level among the confidence levels of each candidate object category is lower than or equal to a preset confidence threshold.

[0161] If the object category classification conditions are met, the target object category is determined based on the first identification result;

[0162] If the object category classification conditions are not met, the first object category prediction result is adjusted based on the first identification result, and the target object category is determined based on the adjusted first object category prediction result.

[0163] In some embodiments, the target element includes a first element corresponding to a first object category and a second element corresponding to a second object category;

[0164] When the identification module 42 determines the category of the target object based on the first identification result, it performs the following steps:

[0165] If the first recognition result indicates that the first feature image contains the first element, then the target object category is determined to be the first object category;

[0166] If the first recognition result indicates that the first feature image does not contain the second element, the target object category is determined to be the second object category.

[0167] In some embodiments, when the identification module 42 adjusts the first object category prediction result based on the first identification result and determines the target object category based on the adjusted first object category prediction result, it performs the following steps:

[0168] Based on the first identification result, the preferred object category corresponding to the first object is determined from a plurality of candidate object categories; the preferred object category is used to indicate that the probability of the target object category being the corresponding candidate object category is the highest.

[0169] Increase the confidence level of the first object category prediction result corresponding to the preferred object category by a preset amount to obtain the adjusted confidence level;

[0170] Based on the adjusted confidence level, the candidate object category with the highest confidence level is determined as the target object category.

[0171] In some embodiments, the device 40 further includes:

[0172] The third acquisition module is used to acquire sample data before acquiring the first feature image and first text information of the first object; the sample data includes a sample feature image, sample text information, and tag information of the sample object; the tag information includes element information of the target element included in the sample feature image and an object category identifier of the sample object category; the sample feature image includes a sample object attribute image for representing the object attribute information of the sample object, and a sample environment image for representing the service environment information of the sample object; the sample text information includes text description information of the sample object, and a task instruction for classifying the object category of the sample object;

[0173] The determination module is used to analyze the sample feature image and the sample text information through a first model to obtain a second object category prediction result for the sample object; to identify the target elements in the sample feature image to obtain a second identification result for the target elements; and to determine the predicted object category of the sample object based on the second object category prediction result and the second identification result.

[0174] The training module is used to iteratively train the first model based on the predicted object category, the second recognition result, and the label information to obtain the multimodal large model.

[0175] In some embodiments, when the training module iteratively trains the first model based on the predicted object category, the second recognition result, and the label information to obtain the multimodal large model, it performs the following steps:

[0176] Based on the predicted object category and the object category identifier, a first loss function of the first model is determined; the first loss function is used to represent the degree of difference between the predicted object category and the sample object category.

[0177] Based on the second identification result and the feature information, a second loss function of the first model is determined; the second loss function is used to represent the degree of difference between the second identification result and the feature information.

[0178] Based on the first loss function and the second loss function, the target loss function of the first model is determined, and the model parameters of the first model are adjusted according to the target loss function to obtain the adjusted first model;

[0179] Based on the sample data, the model parameters of the adjusted first model are adjusted in the next round until the first model meets the preset iteration termination condition, thus obtaining the multimodal large model.

[0180] In some embodiments, when the training module analyzes the sample feature image and the sample text information using the first model to obtain a second object category prediction result for the sample object, it performs the following steps:

[0181] Visual feature information of the sample feature image is extracted, and the sample text information is preprocessed to obtain preprocessed sample text information;

[0182] The visual feature information and the preprocessed sample text information are fused using the cross-modal attention mechanism of the first model to obtain fused multimodal feature information.

[0183] The multimodal feature information is analyzed based on the task instructions in the sample text information to obtain the second object category prediction result.

[0184] According to the apparatus of this application embodiment, a first feature image and first text information of a first object are acquired, and the first feature image and first text information are input into a pre-trained multimodal large model. The multimodal large model analyzes the first feature image and first text information to obtain a first object category prediction result for the first object; and identifies target elements in the first feature image to obtain a first identification result for the target elements; and then determines the target object category of the first object based on the first object category prediction result and the first identification result. It can be seen that this application, through an end-to-end multimodal reasoning process, not only utilizes a multimodal large model to perform deep fusion reasoning on image visual information and text information, achieving the effect of automatically and accurately identifying object categories from multi-source heterogeneous data of images and text, avoiding misjudgments based on hard rules, but also, by combining the first identification result of the object attribute image and the environment image of the first object, since the first identification result is used to indicate whether the first feature image contains target elements, the target elements contained in the first feature image can provide reliable data support for the finally identified object category, improving the accuracy and reliability of object category identification.

[0185] Figure 5 A block diagram of an object category display device based on a multimodal large model provided in an embodiment of this application is shown, such as... Figure 5 As shown, the object category display device 50 based on a multimodal large model includes:

[0186] The second acquisition module 51 is used to acquire the object category identification result of the first object; the object category identification result includes the target object category; the object category identification result is obtained according to the object category identification method based on a multimodal large model described in any of the above embodiments;

[0187] The display module 52 is used to display the object category recognition result at the display position of the first object.

[0188] In some embodiments, when the display module 52 displays the object category recognition result at the display position of the first object, it performs the following steps:

[0189] At a designated location in the object attribute image of the first object, an object category identifier corresponding to the target object category is displayed; the display location includes the designated location.

[0190] And / or,

[0191] In the display area of ​​the object attribute information of the first object, the feature information of the target elements contained in the first feature image of the first object is displayed; the display location includes the display area; the object category recognition result also includes the feature information.

[0192] The functions of each module in each device in the embodiments of this application can be found in the corresponding description in the above method, and they have corresponding beneficial effects, which will not be repeated here.

[0193] Figure 6 This is a block diagram for implementing the electronic device provided in the embodiments of this application. Figure 6 As shown, the electronic device includes a memory 601 and a processor 602. The memory 601 stores a computer program that can run on the processor 602. When the processor 602 executes the computer program, it implements the method described in the above embodiments. The number of memories 601 and processors 602 can be one or more. In a specific implementation, the electronic device may also include a communication interface 603 for communicating with external devices and exchanging data.

[0194] In practical implementation, if the memory 601, processor 602, and communication interface 603 are implemented independently, they can be interconnected via a bus to communicate with each other. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0195] Optionally, in a specific implementation, if the memory 601, processor 602 and communication interface 603 are integrated on a single chip, the memory 601, processor 602 and communication interface 603 can communicate with each other through an internal interface.

[0196] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method provided in this application.

[0197] This application provides a computer program product, including a computer program that, when executed by a processor, implements the method provided in this application.

[0198] This application also provides a chip including a processor for calling and executing instructions stored in a memory, causing a communication device with the chip installed to perform the method provided in this application.

[0199] This application also provides a chip, including: an input interface, an output interface, a processor, and a memory. The input interface, output interface, processor, and memory are connected through an internal connection path. The processor is used to execute code in the memory. When the code is executed, the processor is used to execute the method provided in the application embodiment.

[0200] It should be understood that the aforementioned processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. It is worth noting that the processor can be a processor supporting the Advanced Reduced Instruction Set Computing (ARM) architecture.

[0201] Further, optionally, the aforementioned memory may include read-only memory and random access memory. The memory may be volatile memory or non-volatile memory, or may include both. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available. Examples include Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Sync Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0202] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0203] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0204] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0205] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process. Furthermore, the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functionality involved.

[0206] The logic and / or steps described in the flowchart or otherwise herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0207] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. All or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware, the program being stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiments.

[0208] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0209] The above description is merely an exemplary embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope described in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for object category recognition based on a multimodal large model, characterized in that, include: Acquire a first feature image and first text information of a first object; the first feature image includes an object attribute image representing object attribute information of the first object, and an environment image representing service environment information of the first object; the first text information includes text description information of the first object, and a task instruction for classifying the object category of the first object; The first feature image and the first text information are analyzed by a pre-trained multimodal large model to obtain the first object category prediction result of the first object; The target element in the first feature image is identified to obtain a first identification result of the target element; the first identification result is used to indicate whether the first feature image contains the target element. Based on the first object category prediction result and the first identification result, the target object category of the first object is determined; The first object category prediction result includes the confidence level of the target object category as each candidate object category; Determining the target object category of the first object based on the first object category prediction result and the first identification result includes: determining whether a preset object category classification condition is met based on the first object category prediction result and the first identification result. If the object category classification conditions are met, the target object category is determined based on the first identification result; the object category classification conditions include at least one of the following: the first feature image contains the target element, the number of target elements contained in the first feature image is greater than or equal to a preset number threshold, and the highest confidence level among the confidence levels of each candidate object category is lower than or equal to a preset confidence threshold.

2. The method according to claim 1, characterized in that, Determining the target object category of the first object based on the first object category prediction result and the first identification result includes: If the object category classification conditions are not met, the first object category prediction result is adjusted based on the first identification result, and the target object category is determined based on the adjusted first object category prediction result.

3. The method according to claim 2, characterized in that, The target elements include a first element corresponding to a first object category and a second element corresponding to a second object category; Determining the target object category based on the first identification result includes: If the first recognition result indicates that the first feature image contains the first element, then the target object category is determined to be the first object category; If the first recognition result indicates that the first feature image does not contain the second element, the target object category is determined to be the second object category.

4. The method according to claim 2, characterized in that, The step of adjusting the first object category prediction result based on the first identification result and determining the target object category based on the adjusted first object category prediction result includes: Based on the first identification result, the preferred object category corresponding to the first object is determined from a plurality of candidate object categories; the preferred object category is used to indicate that the probability of the target object category being the corresponding candidate object category is the highest. Increase the confidence level of the first object category prediction result corresponding to the preferred object category by a preset amount to obtain the adjusted confidence level; Based on the adjusted confidence level, the candidate object category with the highest confidence level is determined as the target object category.

5. The method according to claim 1, characterized in that, Before acquiring the first feature image and first text information of the first object, the method further includes: Acquire sample data; the sample data includes sample feature images, sample text information, and tag information of sample objects; the tag information includes element information of the target elements included in the sample feature images and object category identifiers of the sample object categories; the sample feature images include sample object attribute images representing the object attribute information of the sample objects, and sample environment images representing the service environment information of the sample objects; the sample text information includes text description information of the sample objects, and task instructions for classifying the object categories of the sample objects; The sample feature image and sample text information are analyzed by a first model to obtain a second object category prediction result for the sample object; the target elements in the sample feature image are identified to obtain a second identification result for the target elements; and the predicted object category of the sample object is determined based on the second object category prediction result and the second identification result. Based on the predicted object category, the second identification result, and the label information, the first model is iteratively trained to obtain the multimodal large model.

6. The method according to claim 5, characterized in that, The step of iteratively training the first model based on the predicted object category, the second identification result, and the label information to obtain the multimodal large model includes: Based on the predicted object category and the object category identifier, a first loss function of the first model is determined; the first loss function is used to represent the degree of difference between the predicted object category and the sample object category. Based on the second identification result and the feature information, a second loss function of the first model is determined; the second loss function is used to represent the degree of difference between the second identification result and the feature information. Based on the first loss function and the second loss function, the target loss function of the first model is determined, and the model parameters of the first model are adjusted according to the target loss function to obtain the adjusted first model; Based on the sample data, the model parameters of the adjusted first model are adjusted in the next round until the first model meets the preset iteration termination condition, thus obtaining the multimodal large model.

7. The method according to claim 5, characterized in that, The step of analyzing the sample feature image and the sample text information using a first model to obtain the second object category prediction result for the sample object includes: Visual feature information of the sample feature image is extracted, and the sample text information is preprocessed to obtain preprocessed sample text information; The visual feature information and the preprocessed sample text information are fused using the cross-modal attention mechanism of the first model to obtain fused multimodal feature information. The multimodal feature information is analyzed based on the task instructions in the sample text information to obtain the second object category prediction result.

8. A method for displaying object categories based on a multimodal large model, characterized in that, include: Obtain the object category identification result of the first object; The object category identification result includes the target object category; The object category identification result is obtained by the object category identification method based on a multimodal large model according to any one of claims 1 to 7; The object category identification result is displayed at the display position of the first object.

9. The method according to claim 8, characterized in that, The step of displaying the object category recognition result at the display position of the first object includes: At a designated location in the object attribute image of the first object, an object category identifier corresponding to the target object category is displayed; the display location includes the designated location. And / or, In the display area of ​​the object attribute information of the first object, the feature information of the target elements contained in the first feature image of the first object is displayed; the display location includes the display area; the object category recognition result also includes the feature information.

10. An object category recognition device based on a multimodal large model, characterized in that, include: The first acquisition module is used to acquire a first feature image and first text information of a first object; the first feature image includes an object attribute image representing object attribute information of the first object and an environment image representing service environment information of the first object; the first text information includes text description information of the first object and a task instruction for classifying the object category of the first object. The recognition module is used to analyze the first feature image and the first text information through a pre-trained multimodal large model to obtain the first object category prediction result of the first object; The target element in the first feature image is identified to obtain a first identification result of the target element; the first identification result is used to indicate whether the first feature image contains the target element. Based on the first object category prediction result and the first identification result, the target object category of the first object is determined; The first object category prediction result includes the confidence level of the target object category as each candidate object category; the recognition module is further configured to: determine whether a preset object category classification condition is met based on the first object category prediction result and the first recognition result; if the object category classification condition is met, determine the target object category based on the first recognition result; the object category classification condition includes at least one of the following: the first feature image contains the target element, the number of target elements contained in the first feature image is greater than or equal to a preset number threshold, and the highest confidence level among the confidence levels of each candidate object category is lower than or equal to a preset confidence threshold.

11. An object category display device based on a multimodal large model, characterized in that, include: The second acquisition module is used to acquire the object category recognition result of the first object; The object category identification result includes the target object category; The object category identification result is obtained by the object category identification method based on a multimodal large model according to any one of claims 1 to 7; The display module is used to display the object category recognition result at the display position of the first object.

12. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory, wherein the processor, when executing the computer program, implements the method of any one of claims 1 to 9.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 9.

14. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 9.

Images