Target object attribute information prediction method, electronic equipment and computer storage medium

Abstract

The invention discloses a target object attribute information prediction method, electronic equipment and a computer storage medium. The method comprises the following steps: acquiring a collected image and prediction instruction data; determining target prompt information of a target object in the acquired image according to the acquired image and the prediction instruction data; and inputting the target prompt information, the acquired image and the prediction instruction data into the multi-modal large model to obtain attribute information of the target object predicted by the multi-modal large model. Therefore, the credibility of attribute information prediction is improved.

Description

Technical Field

[0001] The present invention relates to the field of machine vision technology, and in particular to a target object attribute information prediction method, electronic equipment and computer storage medium. Background Art

[0002] Object attribute information estimation technology has shown broad application prospects in various fields, including market analysis and marketing, personalized recommendations, entertainment and interaction, medical and health, academic research, intelligent transportation, financial services and education. At present, attribute values ​​are usually estimated only for object samples, which leads to the problem of single attribute value estimation and low reliability.

[0003] Based on this, how to predict highly reliable attribute information becomes a technical problem that needs to be solved urgently. Summary of the invention

[0004] The main technical problem solved by the present application is to provide a method for predicting attribute information of a target object, an electronic device and a computer-readable storage medium, which can improve the credibility of attribute information prediction.

[0005] To solve the above technical problems, a technical solution adopted in the present application is: a method for predicting target object attribute information is provided, and the method for predicting target object attribute information includes: obtaining a captured image and problem description information of the attribute to be predicted of the target object in the captured image; determining target prompt information of the target object in the captured image based on the captured image and the problem description information; inputting the target prompt information, the captured image and the problem description information into a multimodal large model to obtain the attribute information of the target object predicted by the multimodal large model.

[0006] In one embodiment, the multimodal large model includes a first sub-model and a second sub-model, and the step of inputting the target prompt information, the acquired image, and the question description information into the multimodal large model to obtain the attribute information of the target object predicted by the multimodal large model includes: inputting the acquired image and the question description information into the first sub-model to obtain the word-unit information predicted by the first sub-model; and inputting the word-unit information, the question description information, and the target prompt information into the second sub-model to obtain the attribute information of the target object predicted by the second sub-model.

[0007] In one embodiment, the first sub-model includes a visual encoder and a text decoder, and the step of inputting the acquired image and the question description information into the first sub-model to obtain word unit information predicted by the first sub-model includes: encoding the acquired image by the visual encoder to obtain a visual feature vector; performing text decoding on the question description information and the visual feature vector by the text decoder to obtain a text feature vector; and determining the word unit information based on the visual feature vector and the text feature vector.

[0008] In one embodiment, the word-gram information includes context words and content words, and the step of determining the word-gram information based on the visual feature vector and the text feature vector includes: compressing the text feature vector and the visual feature vector to obtain a compressed feature vector; converting the compressed feature vector to obtain the context word-gram; converting the visual feature vector to obtain the content word-gram.

[0009] In one embodiment, the step of inputting the target prompt information, the acquired image and the problem description information into the multimodal large model to obtain the attribute information of the target object predicted by the multimodal large model includes: obtaining the facial posture angle information and / or the facial occlusion information in the target prompt information and the age problem in the problem description information; inputting the facial posture angle information and / or the facial occlusion information, the acquired image and the age problem into the multimodal large model to obtain the age information of the target object predicted by the multimodal large model.

[0010] In one embodiment, the question description information includes question text, and the step of determining target prompt information of the target object in the acquired image based on the acquired image and the question description information includes: performing feature extraction on the acquired image to obtain attribute prior information of the target object in the acquired image; and determining the target prompt information based on the attribute prior information and the question text.

[0011] In one embodiment, the step of determining the target prompt information based on the attribute prior information and the question text includes: determining a target prompt template from a preset prompt template based on the question text; obtaining the value of each attribute in the attribute prior information; and filling each attribute value into the target prompt template to obtain the target prompt information.

[0012] In one embodiment, the method also includes: obtaining a sample image, prediction instruction data of a sample object in the sample image, and prompt information about the sample object, the prediction instruction data including question description information of an attribute to be predicted of the sample object and answer information of the question description information; inputting the sample image, the prediction instruction data of the sample object, and the prompt information of the sample object into a multimodal large model to be trained to obtain an attribute information prediction result output by the multimodal large model to be trained; adjusting the multimodal large model according to the attribute information prediction result, the prediction instruction data of the sample object, and the prompt information of the sample object until a multimodal large model that meets the requirements is obtained.

[0013] To solve the above technical problems, another technical solution adopted in the present application is: to provide an electronic device, including a memory and a processor, wherein the memory stores program instructions, and the processor retrieves the program instructions from the memory to execute the above target object attribute information prediction method.

[0014] In order to solve the above technical problems, another technical solution adopted by the present application is: providing a computer-readable storage medium including program data stored therein, and the program data is used to implement the above target object attribute information prediction method when executed by a processor.

[0015] The above scheme determines the target prompt information of the target object in the acquired image based on the acquired image and the attribute description information of the target object's attribute to be predicted in the acquired image; inputs the target prompt information, the acquired image and the attribute description information into the multimodal large model to obtain the attribute information of the target object predicted by the multimodal large model. When the multimodal large model predicts the attribute information of the target object, it combines the acquired image, the attribute description information of the target object's attribute to be predicted in the acquired image and the target prompt information, thereby improving the accuracy of the attribute information prediction and the credibility of the attribute information prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work, among which: Figure 1 It is a flowchart of an exemplary embodiment of a method for predicting target object attribute information shown in the present application; Figure 2 yes Figure 1 FIG. 1 is a flow chart of an exemplary embodiment of step S130 in the target object attribute information prediction method; Figure 3 yes Figure 1 FIG. 1 is a flow chart of an exemplary embodiment of step S120 in the target object attribute information prediction method; Figure 4 It is a structural schematic diagram of an exemplary embodiment of a multi-modal large model shown in the present application; Figure 5 is a schematic structural diagram of an exemplary embodiment of an attribute information prediction device shown in the present application; Figure 6 It is a structural schematic diagram of an embodiment of an electronic device provided by the present application; Figure 7 It is a structural schematic diagram of an embodiment of a computer-readable storage medium provided by the present application. DETAILED DESCRIPTION

[0017] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It will be appreciated that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only some but not all structures related to the present application are shown in the drawings. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the art without making creative work are within the scope of protection of the present application.

[0018] For details, please refer to Figure 1 , Figure 1 It is a flowchart of an exemplary embodiment of a method for predicting target object attribute information shown in the present application.

[0019] The execution subject of the target object attribute information prediction method may be a terminal device or a server or other processing device, wherein the terminal device may be a user equipment (UE), a computer, a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The execution subject of the target object attribute information prediction method may also be an attribute information prediction device. In some possible implementations, the target object attribute information prediction method may be implemented by a processor calling a computer-readable instruction stored in a memory.

[0020] In the embodiment of the present application, the attribute information prediction device is used as the execution subject for description. Specifically, the target object attribute information prediction method of this embodiment includes the following steps: S110, obtaining a captured image and problem description information of an attribute to be predicted of a target object in the captured image.

[0021] The target objects can be animals, plants, humans, etc.

[0022] The attributes to be predicted can be age, gender, posture, etc.

[0023] The question description information may be determined by the attribute information prediction device in response to the user's input. As an exemplary embodiment, the attribute information prediction device receives voice information input by the user, and the attribute information prediction device analyzes the voice information to obtain the question description information. As another exemplary embodiment, the attribute information prediction device receives text information input by the user and uses the text information as the question description information. It should be noted that the question description information is also the user's question about the attributes of the target object in the captured image or the user's question about the attributes of the target object in the captured image. For example, if the attribute to be predicted is posture, the question description information may be, What is the posture of the target object in the captured image? If the attribute to be predicted is age, the question description information may be, What is the age of the target object in the captured image? If the attribute to be predicted is gender, the question description information may be, What is the gender of the target object in the captured image? S120, determining target prompt information of the target object in the collected image according to the collected image and the question description information.

[0024] The target prompt information is used to guide the multimodal large model to use or partially use the target prompt information when making predictions, so as to improve the prediction accuracy and credibility.

[0025] As an example, the attribute information prediction device can input the captured image and the question description information into the trained prompt model to obtain the target prompt information of the target object in the captured image output by the prompt model. For example, if the attribute to be predicted is age, the question description information can be age-related information, specifically what is the age of the target object in the captured image? The target prompt information can be age-related information, specifically facial posture angle information or facial occlusion information of the target object. It should be noted that the facial posture angle information can be the specific degree of the upward posture angle of the face. For another example, if the attribute to be predicted is gender, the question description information can be gender-related information, specifically what is the gender of the target object in the captured image? The target prompt information can be gender-related information, specifically physical feature information such as the hair length or Adam's apple of the target object.

[0026] S130, inputting the target prompt information, the collected image and the problem description information into the multimodal large model to obtain the attribute information of the target object predicted by the multimodal large model.

[0027] The attribute information may be the target object's gender, age, possible age range, posture information, information prediction credibility, etc. It should be noted that the attribute information includes not only the attribute information of the attribute to be predicted, but also other attribute information related to the attribute to be predicted. For example, if the attribute to be predicted is age, the predicted attribute information may be the possible age range of the target object, the impossible age range, the local occlusion degree of the target object, the posture angle of the target object, and other information.

[0028] The attribute information prediction device inputs the target prompt information, the collected image, and the problem description information into the multimodal large model to obtain the attribute information of the target object predicted by the multimodal large model. As an example, the attribute information prediction device obtains the facial posture angle information and / or facial occlusion information in the target prompt information and the age problem in the problem description information, inputs the facial posture angle information and / or facial occlusion information, the collected image, and the age problem into the multimodal large model, and obtains the age information of the target object predicted by the multimodal large model.

[0029] It can be seen that the target object attribute information prediction method of the embodiment of the present application determines the target prompt information of the target object in the acquired image according to the acquired image and the attribute description information of the target object's attribute to be predicted in the acquired image; the target prompt information, the acquired image and the attribute description information are input into the multimodal large model to obtain the attribute information of the target object predicted by the multimodal large model. When the multimodal large model predicts the attribute information of the target object, it combines the acquired image, the attribute description information of the target object's attribute to be predicted in the acquired image and the target prompt information, thereby improving the accuracy of the attribute information prediction and the credibility of the attribute information prediction.

[0030] Based on the above embodiments, the embodiments of the present application adopt Figure 2 The flowchart details how the multimodal large model predicts the attribute information of the target object in the collected image based on the target prompt information, the collected image and the problem description information. Figure 2 , Figure 2 yes Figure 1 The flowchart of an exemplary embodiment of step S130 in the target object attribute information prediction method is shown. Specifically, step S130 inputs the target prompt information, the collected image and the problem description information into the multimodal large model, and the process of obtaining the attribute information of the target object predicted by the multimodal large model includes the following steps: S210, inputting the collected image and question description information into the first sub-model to obtain word-unit information predicted by the first sub-model.

[0031] The multimodal large model includes a first sub-model and a second sub-model, wherein the first sub-model is used to extract word-unit information from the captured image and question description information input therein.

[0032] The attribute information prediction device processes the captured image and the question description information through the first sub-model to obtain the word-unit information predicted by the first sub-model. Exemplarily, the first sub-model includes a visual encoder and a text decoder. The attribute information prediction device encodes the captured image through the visual encoder to obtain a visual feature vector; performs text decoding on the question description information and the visual feature vector through the text decoder to obtain a text feature vector; and then determines the word-unit information based on the visual feature vector and the text feature vector. It should be noted that the visual feature vector can be represented as X, , N is the number of image blocks in the captured image, C is the number of feature channels, and the text feature vector can be expressed as Q, , M represents the number of text feature vectors. It should be further explained that the text decoder is mainly used for cross-modal interaction between images and texts, and the text decoder in the embodiment of the present application may be a Q-Former structure.

[0033] Regarding the method of encoding the captured image through a visual encoder to obtain a visual feature vector, the attribute information prediction device extracts features from the target object region in the captured image through a visual encoder to obtain a visual feature vector. The attribute information prediction device may use a YOLOv5 model to obtain the target object region in the captured image. It should be noted that the captured image can be represented as , H represents the height of the acquired image, W represents the width of the acquired image, and the area where the target object is located in the acquired image can be expressed as , the visual feature vector can be expressed as , C represents the number of feature channels. Among them, , N represents the number of image blocks, and P represents the image block size.

[0034] Regarding the method of determining word-meta information based on the visual feature vector and the text feature vector, as an example, the attribute information prediction device can fuse the visual feature vector and the text feature vector to obtain word-meta information. As another example, the attribute information prediction device can construct a context attention module, and process the visual feature vector and the text feature vector through the context attention module to aggregate the visual feature vector related to the text to obtain word-meta information. Specifically, the attribute information prediction device compresses the text feature vector and the visual feature vector to obtain a compressed feature vector; converts the compressed feature vector to obtain a context word-meta; and converts the visual feature vector to obtain a content word-meta. It should be noted that regarding the method of converting the visual feature vector to obtain the content word-meta, the attribute information prediction device can convert the visual feature vector through adaptive pooling and a multi-layer perceptron (MLP) projector to obtain the content word-meta. Regarding the method of converting the compressed feature vector to obtain the context word-metaphor, the attribute information prediction device can convert the compressed feature vector through a multi-layer perceptron (MLP) projector to obtain the context word-metaphor, so as to achieve alignment between the content word-metaphor and the context word-metaphor.

[0035] Specifically, the compressed feature vector satisfies the following formula:

[0036] Among them, E represents the compressed feature vector, Mean() represents the mean function, Soft max() represents the normalized exponential function, Q represents the text feature vector, represents the transpose of the visual feature vector, and X represents the visual feature vector.

[0037] S220, input word-unit information, question description information and target prompt information into the second sub-model to obtain attribute information of the target object predicted by the second sub-model.

[0038] The second sub-model may be an LLM (Large Language Model) model.

[0039] It can be seen that the target object attribute information prediction method of the embodiment of the present application inputs the collected image and question description information into the first sub-model to obtain the word-unit information predicted by the first sub-model; the word-unit information, question description information and target prompt information are input into the second sub-model to obtain the attribute information of the target object predicted by the second sub-model. Thus, the attribute information is predicted based on the question description information and the target prompt information. Compared with only predicting the attribute value, it can alleviate the learning bias caused by network prediction based on the simple attribute value, and improve the credibility of the attribute information prediction.

[0040] Based on the above embodiments, the embodiments of the present application adopt Figure 3 The flowchart details how to determine the target prompt information, please refer to Figure 3 , Figure 3 yes Figure 1 FIG. 1 is a flow chart of an exemplary embodiment of step S120 in the target object attribute information prediction method. Specifically, the process of determining the target prompt information of the target object in the collected image according to the collected image and the question description information in step S120 includes the following steps: S310, extracting features from the captured image to obtain prior information on the attributes of the target object in the captured image.

[0041] The attribute prior information GPO (Gender, Pose, Occlusion) includes the gender, head posture, occlusion status, etc. of the target object in the captured image.

[0042] As an example, the attribute information prediction device may input the captured image into the FaceXFormer expert model to obtain the attribute prior information of the target object in the captured image. As another example, the attribute information prediction device may compare the features in the captured image with the preset features to determine the attribute prior information. Specifically, if the similarity between the features in the captured image and the preset features is greater than the preset similarity threshold, the attribute information prediction device determines the attribute prior information of the target object based on the attributes corresponding to the preset features and the feature values ​​in the captured image.

[0043] S320, determining target prompt information according to the attribute prior information and the question text.

[0044] As an example, the attribute information prediction device determines the target prompt template from the preset prompt template according to the question text; obtains the attribute values ​​in the attribute prior information; fills the attribute values ​​into the target prompt template accordingly to obtain the target prompt information. As another example, the attribute information prediction device determines the target prompt information according to the matching degree between the question text and the attribute prior information. For example, the question text is "What is the age of the target object in the captured image?" The attribute prior information includes an estimated age of 15 years old, an estimated height of 150, and an estimated gender of male. It can be determined that the attribute prior information of the estimated age of 15 years old matches the question text, and the corresponding estimated age of 15 years old is determined as the target prompt information.

[0045] It should be noted that after determining the target prompt information of the target object in the captured image, the attribute information prediction device can determine the credibility of the attribute information estimation according to the target prompt information. Exemplarily, the attribute information prediction device can compare the target prompt information with the preset information to determine the credibility of the attribute information estimation. For example, if the target prompt information is that the target object posture angle is 60 degrees and the occlusion of the target part exceeds two-thirds, then it is determined whether the target object posture angle 60 degrees is greater than 45 degrees and whether the occlusion of the target part exceeds one-third. If the posture angle is greater than 45 degrees and the occlusion state of the target part exceeds one-third, it is determined that the prediction credibility of the corresponding attribute information is not high and there is a risk of inaccuracy; if the posture angle is less than or equal to 45 degrees and the occlusion state of the target part exceeds one-third, it is determined that the prediction credibility of the corresponding attribute information is high; if the posture angle is greater than 45 degrees and the occlusion state of the target part does not exceed one-third, it is determined that the prediction credibility of the corresponding attribute information is high. If the posture angle is greater than 45 degrees and the occlusion state of the target part does not exceed one-third, it is determined that the prediction credibility of the corresponding attribute information is high.

[0046] It can be seen that the target object attribute information prediction method of the embodiment of the present application extracts features from the captured image to obtain the attribute prior information of the target object in the captured image; and determines the target prompt information based on the attribute prior information and the question text. The target prompt information can be determined based on the attribute prior information, and then the model is guided to selectively use the target prompt information based on the determined target prompt information, which improves the accuracy and prediction credibility of the attribute information predicted by the multimodal large model.

[0047] On the basis of the above embodiment, in order to obtain a multimodal large model with accurate prediction, the attribute information prediction device trains the multimodal large model before predicting the attribute information of the target object. Specifically, the attribute information prediction device obtains the sample image, the prediction instruction data of the sample object in the sample image, and the prompt information about the sample object, and the prediction instruction data includes the question description information of the attribute to be predicted of the sample object and the answer information of the question description information; the sample image, the prediction instruction data of the sample object and the prompt information of the sample object are input into the multimodal large model to be trained to obtain the attribute information prediction result output by the multimodal large model to be trained; the multimodal large model is adjusted according to the attribute information prediction result, the prediction instruction data of the sample object and the prompt information of the sample object until a multimodal large model that meets the requirements is obtained. It should be noted that the embodiment of the present application can use LoRA (Low-Rank Adaptation) to perform model training fine-tuning processing. Specifically, the attribute information prediction device obtains the information difference between the attribute information prediction result and the answer information of the sample object and the prompt information of the sample object; the weight parameter in the first sub-model is adjusted according to the information difference until the attribute information prediction result output by the second sub-model meets the requirements, and a multimodal large model that meets the requirements is obtained.

[0048] Regarding the method of obtaining the prediction instruction data of the sample object in the sample image, the attribute information prediction device can use the Qwen2-VL multimodal model to generate prediction instruction data related to the attribute, that is, question-answer dialogue instruction data. Specifically, the attribute information prediction device inputs the captured image and the question description information about the target object in the captured image into the Qwen2-VL multimodal model to obtain the answer information output by the Qwen2-VL multimodal model. For example, the question description information is how old the target object in the captured image is, and the answer information is the predicted age and the possible age range and impossible age range of the target object, thereby enabling the Qwen2-VL multimodal model to be used to expand the prediction of attribute information.

[0049] Regarding the method of obtaining the prompt information of the sample object, the attribute information prediction device can detect and process the sample image through YOLOv5 to obtain the image area where the target object is located; then the image area where the target object is located is input into the FaceXFormer expert model to obtain the attribute prior information; then the problem description information and the attribute prior information are used to guide the multimodal large model to selectively use the attribute prior information during training to obtain the prompt information of the sample object.

[0050] To elaborate on the above embodiment, Figure 4 The following example is used to illustrate this. Figure 4 , Figure 4The framework example diagram of the multimodal large model is shown in . As shown in the figure, the attribute information prediction device inputs the captured image and the age question description information about the target object in the captured image into the multimodal large model, and encodes the captured image through the visual encoder frozen in the multimodal large model to obtain the visual feature vector X; the age question description information "Question: How old is the face in the picture?" and the visual feature vector X are text-decoded through the text decoder in the multimodal large model to obtain the text feature vector Q, that is, the text query; the visual feature vector X and the text feature vector Q are compressed through the context attention module in the multimodal large model to obtain the compressed feature vector E; the compressed feature vector E is processed by the projector to obtain the above The following word, also known as the context token; the visual feature vector X is pooled to obtain a pooled visual feature vector, and the pooled visual feature vector X is processed by a projector to obtain a content word, also known as a content token; then the content word, context word, age question description information "Question: What is the age of the face in the picture?" and the target prompt information determined based on the question description information, that is, the manual prompt word in the figure, are input into the LLM model fine-tuned based on LoRA to obtain attribute information about the target object, such as "answer: The age in the picture is 40~50 years old, the gender is male, there is no occlusion, there is no posture angle exceeding the threshold, and the age prediction value has high credibility" in the figure.

[0051] See also Figure 5 , Figure 5 It is a structural diagram of an exemplary embodiment of the attribute information prediction device shown in the present application. The attribute information prediction device 500 includes an acquisition module 510, a determination module 520 and a prediction module 530. The acquisition module 510 is used to acquire the acquired image and the problem description information of the attribute to be predicted of the target object in the acquired image; the determination module 520 is used to determine the target prompt information of the target object in the acquired image according to the acquired image and the problem description information; the prediction module 530 is used to input the target prompt information, the acquired image and the problem description information into the multimodal large model to obtain the attribute information of the target object predicted by the multimodal large model.

[0052] In the above scheme, the attribute information prediction device determines the target prompt information of the target object in the acquired image based on the acquired image and the attribute description information of the target object's attribute to be predicted in the acquired image; the target prompt information, the acquired image and the attribute description information are input into the multimodal large model to obtain the attribute information of the target object predicted by the multimodal large model. When the multimodal large model predicts the attribute information of the target object, it combines the acquired image, the attribute description information of the target object's attribute to be predicted in the acquired image and the target prompt information, thereby improving the accuracy of the attribute information prediction and the credibility of the attribute information prediction.

[0053] The functions of each module can be found in the target object attribute information prediction method embodiment, which will not be repeated here.

[0054] In order to implement the target object attribute information prediction method of the above embodiment, the present application proposes another electronic device, which is specifically referred to as Figure 6 , Figure 6 It is a structural schematic diagram of an embodiment of an electronic device provided by the present application.

[0055] The electronic device 600 includes a memory 601 and a processor 602 , wherein the memory 601 and the processor 602 are coupled.

[0056] The memory 601 is used to store program data, and the processor 602 is used to execute the program data to implement the target object attribute information prediction method of the above embodiment.

[0057] In this embodiment, the processor 602 may also be referred to as a CPU (Central Processing Unit). The processor 602 may be an integrated circuit chip having signal processing capabilities. The processor 602 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. A general-purpose processor may be a microprocessor or the processor 602 may also be any conventional processor, etc.

[0058] The present application also provides a computer-readable storage medium, such as Figure 7 As shown, the computer-readable storage medium 700 is used to store program data 701. When the program data 701 is executed by the processor, it is used to implement the target object attribute information prediction method in the method embodiment of the present application.

[0059] The method involved in the embodiment of the target object attribute information prediction method of the present application, when implemented in the form of a software functional unit and sold or used as an independent product, can be stored in a device, such as a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or part of the contribution to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0060] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for predicting target object attribute information, characterized in that: The target object attribute information prediction method comprises: Acquire a captured image and problem description information of a property to be predicted of a target object in the captured image; Determine target prompt information of a target object in the captured image according to the captured image and the problem description information; The target prompt information, the collected image and the problem description information are input into a multimodal large model to obtain the attribute information of the target object predicted by the multimodal large model.

2. The target object attribute information prediction method according to claim 1, characterized in that: The multimodal large model includes a first sub-model and a second sub-model, and the step of inputting the target prompt information, the collected image, and the problem description information into the multimodal large model to obtain the attribute information of the target object predicted by the multimodal large model includes: Inputting the collected image and the question description information into the first sub-model to obtain word-unit information predicted by the first sub-model; The word-unit information, the question description information and the target prompt information are input into the second sub-model to obtain the attribute information of the target object predicted by the second sub-model.

3. The target object attribute information prediction method according to claim 2, characterized in that: The first sub-model includes a visual encoder and a text decoder, and the step of inputting the collected image and the question description information into the first sub-model to obtain word-meta information predicted by the first sub-model includes: Encoding the captured image by the visual encoder to obtain a visual feature vector; Performing text decoding processing on the problem description information and the visual feature vector by the text decoder to obtain a text feature vector; The word-unit information is determined according to the visual feature vector and the text feature vector.

4. The target object attribute information prediction method according to claim 3, characterized in that: The word-meta information includes context word-meta and content word-meta. The step of determining the word-meta information according to the visual feature vector and the text feature vector includes: Compressing the text feature vector and the visual feature vector to obtain a compressed feature vector; Performing conversion processing on the compressed feature vector to obtain the context word unit; The visual feature vector is converted to obtain the content word.

5. The target object attribute information prediction method according to claim 1, characterized in that: The step of inputting the target prompt information, the collected image and the problem description information into the multimodal large model to obtain the attribute information of the target object predicted by the multimodal large model includes: Acquire the facial posture angle information and / or the facial occlusion information in the target prompt information and the age question in the question description information; The facial posture angle information and / or the facial occlusion information, the collected image and the age problem are input into the multimodal large model to obtain the age information of the target object predicted by the multimodal large model.

6. The target object attribute information prediction method according to claim 1, characterized in that: The question description information includes a question text, and the step of determining target prompt information of a target object in the collected image according to the collected image and the question description information includes: Extracting features from the captured image to obtain prior information on the attributes of the target object in the captured image; The target prompt information is determined according to the attribute prior information and the question text.

7. The target object attribute information prediction method according to claim 6, characterized in that: The step of determining the target prompt information according to the attribute prior information and the question text comprises: Determining a target prompt template from preset prompt templates according to the question text; Obtaining the value of each attribute in the attribute prior information; Fill each attribute value into the target prompt template accordingly to obtain the target prompt information.

8. The target object attribute information prediction method according to claim 1, characterized in that: The method further comprises: Acquire a sample image, prediction instruction data of a sample object in the sample image, and prompt information about the sample object, wherein the prediction instruction data includes question description information of a property to be predicted of the sample object and answer information of the question description information; Inputting the sample image, the prediction instruction data of the sample object and the prompt information of the sample object into the multimodal large model to be trained, and obtaining the attribute information prediction result output by the multimodal large model to be trained; The multimodal large model is adjusted according to the attribute information prediction result, the prediction instruction data of the sample object and the prompt information of the sample object until a multimodal large model that meets the requirements is obtained.

9. An electronic device, characterized in that: include: A memory and a processor, wherein the memory stores program instructions, and the processor retrieves the program instructions from the memory to execute the method according to any one of claims 1 to 8.

10. A computer storage medium, characterized in that: include: Program data is stored, and when the program data is executed by a processor, it is used to implement the method according to any one of claims 1 to 8.

Images