A property scene perception and decision-making method, device, equipment and storage medium

Abstract

The present invention discloses a property scene perception and decision method, device, equipment and storage medium, and the method includes: preprocessing and labeling the target detection data set to a standard format to obtain a property data set containing multiple images; aligning the labels of the property data set with the public data set to obtain a common label set; and fusing the common label set through data screening to obtain a fused data set; converting the fused data set into a visual question-answering data set, and integrating the professional knowledge of the property scene to construct a visual-text multimodal data set, and using the visual-text multimodal data set for model training to obtain a multimodal large model; using the pre-trained visual model to fine-tune the multimodal large model; deploying the fine-tuned multimodal large model into the property business system. The present invention solves the problems of high training cost and inability to understand contextual semantics of existing multimodal large models, and improves the practical application value of multimodal large models.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and computer vision technology, and in particular to a property scene perception and decision-making method, device, equipment and storage medium. Background Art

[0002] With the development of artificial intelligence and deep learning technology, computer vision technology has made significant progress in tasks such as object detection and instance segmentation. However, there are many challenges in the actual deployment of these models. First, high-performance computer vision models require a large amount of training data to maintain high performance, and obtaining high-quality, finely labeled data is time-consuming and costly. Second, pure visual models lack the ability to understand contextual semantic information, and understanding the semantic information of the scene is very important in practical applications.

[0003] In order to solve the above problems, multimodal large models have become a research hotspot. However, existing multimodal large models often rely on large and unified training datasets, which makes it difficult for them to flexibly adapt to the differences in data distribution between different application fields. Summary of the invention

[0004] The embodiments of the present invention provide a property scene perception and decision-making method, device, equipment and storage medium, aiming to solve the problems of high training cost and inability to understand contextual semantics in existing multimodal large models.

[0005] In a first aspect, an embodiment of the present invention provides a property scene perception and decision-making method based on a visual-textual multimodal large model, comprising:

[0006] Preprocess and annotate the target detection dataset to obtain a property dataset containing multiple images;

[0007] Aligning the labels of the property dataset with the public dataset to obtain a common label set; and fusing the common label set through data screening to obtain a fused dataset;

[0008] Converting the fused dataset into a visual question answering dataset, integrating professional knowledge of property scenarios, constructing a visual-text multimodal dataset, and using the visual-text multimodal dataset for model training to obtain a multimodal large model;

[0009] Fine-tuning the multimodal large model using a pre-trained visual model;

[0010] Deploy the fine-tuned multimodal large model into the property business system.

[0011] In a second aspect, an embodiment of the present invention provides a property scene perception and decision-making device based on a visual-textual multimodal large model, comprising:

[0012] An acquisition unit, used for preprocessing and annotating the target detection data set to a format specification, and obtaining a property data set containing multiple images;

[0013] A label alignment unit, used to perform label alignment on the property data set and the public data set to obtain a common label set; and to fuse the common label set through data screening to obtain a fused data set;

[0014] A fusion unit, used to convert the fused data set into a visual question answering data set, integrate the professional knowledge of the property scene, construct a visual-text multimodal data set, and use the visual-text multimodal data set to perform model training to obtain a multimodal large model;

[0015] A fine-tuning unit, used to fine-tune the multimodal large model using a pre-trained visual model;

[0016] The deployment unit is used to deploy the fine-tuned multimodal large model into the property business system.

[0017] In a third aspect, an embodiment of the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the property scene perception and decision-making method based on the visual-textual multimodal large model as described in the first aspect is implemented.

[0018] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the processor executes the property scene perception and decision-making method based on the visual-text multimodal large model as described in the first aspect.

[0019] The embodiment of the present invention provides a property scene perception and decision-making method, device, equipment and storage medium. The present invention fine-tunes a multimodal large model by integrating visual data such as target detection data sets and public data sets with textual knowledge of business scenarios, so that it has the ability to understand contextual semantics and general scene perception, and is suitable for intelligent decision-making and perception in fields such as property management. First, in terms of property data set construction, the existing property scene (such as overflowing trash cans, illegal parking of motor vehicles, people falling, etc.) target detection data sets are used to integrate more visual public data sets to enhance the diversity of data samples and ensure data quality and model generalization capabilities. The labels of the target detection and instance segmentation data sets are sorted out to generate visual question and answer data in a format that supports instruction fine-tuning, providing training materials for the multimodal large model. Secondly, in terms of fine-tuning the multimodal large model, the visual question and answer data and the professional knowledge of the property scene are integrated, and the pre-trained visual model is used for fine-tuning, so that the model can understand each business background, realize the perception and decision-making of general scenes, and enhance the practical application value of the multimodal large model. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0021] Figure 1 A flowchart of a property scene perception and decision-making method based on a visual-textual multimodal large model provided by an embodiment of the present invention;

[0022] Figure 2 A sub-process diagram of a property scene perception and decision-making method based on a visual-textual multimodal large model provided by an embodiment of the present invention;

[0023] Figure 3 Another sub-process diagram of a property scene perception and decision-making method based on a visual-textual multimodal large model provided by an embodiment of the present invention;

[0024] Figure 4 Another sub-process diagram of a property scene perception and decision-making method based on a visual-textual multimodal large model provided by an embodiment of the present invention;

[0025] Figure 5 Another sub-process diagram of a property scene perception and decision-making method based on a visual-textual multimodal large model provided by an embodiment of the present invention;

[0026] Figure 6Another sub-process diagram of a property scene perception and decision-making method based on a visual-textual multimodal large model provided by an embodiment of the present invention;

[0027] Figure 7 A schematic block diagram of the overall process of property scene perception and decision-making based on a visual-textual multimodal large model provided in an embodiment of the present invention;

[0028] Figure 8 A schematic diagram of a property scene perception and decision-making device based on a visual-textual multimodal large model provided by an embodiment of the present invention;

[0029] Fig. 9 A schematic diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0032] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0033] It should be further understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0034] See also Figure 1 , Figure 1 A flow chart of a property scene perception and decision-making method based on a visual-textual multimodal large model provided in an embodiment of the present invention, the method comprising steps S101 to S105:

[0035] S101, preprocessing and annotating the target detection data set to a standard format to obtain a property data set containing multiple images;

[0036] In this embodiment, first, the existing object detection datasets of property scenes are collected and integrated. These datasets contain common targets in the property management process, such as personnel, equipment, facilities, scenes, etc. In order to improve the data quality, the object detection dataset is preprocessed, such as data cleaning, noise removal, and correction of label errors. At the same time, it is also necessary to standardize the annotation format of the object detection dataset. The annotation content may include objects, personnel behavior, and environmental status in the property scene, so as to generate a property dataset containing multiple images to ensure the consistency and reliability of the property dataset.

[0037] S102, aligning labels of the property dataset with the public dataset to obtain a common label set; and fusing the common label set through data screening to obtain a fused dataset;

[0038] In this embodiment, the preprocessed property dataset is label aligned with the public dataset, with the goal of finding common labels between the property dataset and the public dataset. These common labels are the basis for subsequent data fusion.

[0039] Through data screening, the common label set is fused to remove redundant information and ensure the accuracy and completeness of the data set. The fused data set contains richer property scene information and labels, which helps to improve the generalization ability of the model.

[0040] In one embodiment, if Figure 2 As shown, in the step S102, the step of aligning the labels of the property dataset with the public dataset to obtain a common label set includes steps S201 to S204:

[0041] S201. Define the label set of the property data set according to the following formula: ;

[0042] S202: Define the label set of the public data set according to the following formula: ;

[0043] S203. According to the label set of the property data set and the label set of the public data set, a label mapping function is established according to the following formula to map the label of the public data set to the label of the property data set to obtain a valid property label: ,in, express The i-th label in , i∈[1,n], n is a natural number; express The jth label in , j∈[1,m], m is a natural number; ” means that the labels of the public dataset are not mapped to the labels of the property dataset;

[0044] S204. According to the valid property tags, a common tag set is obtained according to the following formula: .

[0045] In this embodiment, in step S201 and step S202, before integrating the property dataset and the public dataset, the label sets of the two need to be aligned to unify the annotation standards and facilitate subsequent data fusion and model training. Therefore, the label set of the property dataset and the label set of the public dataset are shown in the above formula.

[0046] In step S203, a label mapping function is established based on the label set of the property data set and the label set of the public data set. ,Converting the annotations of the public dataset into property labels can achieve label alignment between the public dataset and the property dataset, making the two datasets consistent or comparable at the label level, which is helpful for subsequent tasks such as data fusion and model training.

[0047] In one embodiment, if Figure 3 As shown, in the step S102, the step of fusing the common label set by data screening to obtain a fused data set also includes steps S401 to S406:

[0048] S401, through the property data set Train to get an original model ;

[0049] S402, through the original model The images in the public dataset are inferred and the prediction results are obtained according to the following formula: ,in, is the i-th image, The original model Parameters;

[0050] S403: annotate the public data set Mapped to the labels of the property dataset by the label mapping function: ;

[0051] S404: Based on the real annotation of the public data set , filter out the annotations of the common tag set: ;

[0052] S405: Labeling based on the prediction result and the common label set , calculate the average precision of each image according to the following formula ,in, represents the interpolation accuracy at the recall level r, R represents the set of recall values ​​from 0 to 1 with an increment of 0.1, and K represents the number of different recall points used for calculation in the recall rate R;

[0053] S406: sort all images according to the prediction result and the annotation of the common label set according to the average precision of each image. Sort from high to low to get a sorted list , where M means there are M average precisions in the sorted list , the filtered fusion data set is obtained according to the following formula , where C represents the number of images filtered out.

[0054] In this embodiment, in step S401, since both the public dataset and the property dataset have domain shift problems, a training data screening mechanism based on the image-wise average precision ranking method is proposed. The process is briefly described as follows: based on the source domain and target domain The relationship found between , from the source domain Select a subset of The optimization process can be expressed as follows: in, Defined as being within budget Next, use the subset Trained detection model performance.

[0055] Furthermore, after label alignment is completed, the DINO model is used on the property dataset Train on it to get the original target detection model The loss function of the training process is: ,in, is the original object detection model Parameters, It is a property data set The number of images in is the i-th image, is the corresponding annotation information, It is the loss function for object detection, a weighted sum of the cross entropy loss and the bounding box regression loss.

[0056] Furthermore, according to the following formula, by minimizing the loss function , get the trained original target detection model : .

[0057] In step S402, according to the label mapping function , using the original target detection model When reasoning and filtering public data sets, we only need to focus on the common label set. , used to infer the images in the public dataset and obtain the prediction results. The prediction results will be used to compare with the true annotations of the public dataset to evaluate the original object detection model performance.

[0058] In step S403, the real annotations of the public dataset are Map it to a space or set through the label mapping function. Since the public dataset and the property dataset may use different labeling systems, it is necessary to convert the real annotations of the public dataset into labels compatible with the property dataset through the label mapping function to ensure that the subsequent steps can correctly compare the prediction results and the real annotations.

[0059] In step S404, the annotations in the common tag set are filtered out according to the real annotations of the public data set (after mapping), and the tags that exist in both the public data set and the property data set are filtered out from the common tag set.

[0060] In step S405, the average precision is a metric for evaluating model performance that considers the accuracy of the model at different recall rates. In this step, the average precision of each image is calculated using interpolated precision. This step helps understand the performance of the model in identifying each category.

[0061] In step S406, all images are sorted according to the average precision Sort from high to low, and filter out the first N images as the filtered fusion data set , fusion dataset The fused dataset contains images and their corresponding annotation information. In this step, the fused dataset will contain images that are clearly annotated in both the public dataset and the property dataset and for which the model performs best.

[0062] S103, converting the fused data set into a visual question answering data set, integrating professional knowledge of property scenarios, constructing a visual-text multimodal data set, and using the visual-text multimodal data set to perform model training to obtain a multimodal large model;

[0063] In this embodiment, the fused dataset needs to be converted into a Visual Question Answering (VQA) dataset, which means that a series of questions need to be designed for each image and corresponding answers are given. These questions and answers will be built around the objects, scenes, time in the image, and the relationship between them. In the process of building a visual question answering dataset, it is also necessary to incorporate professional knowledge of property scenes. Professional knowledge includes understanding various aspects of property management, such as facility maintenance, safety management, environmental cleaning, etc., and the possible manifestations of these aspects in images. After the construction of the visual question answering dataset is completed, it is combined with the images in the fused dataset to form a visual-text multimodal dataset. The last step is to use the constructed visual-text multimodal dataset to train a large multimodal model. The large multimodal model can process image and text information at the same time, understand the relationship between them, and give accurate answers or predictions.

[0064] In one embodiment, if Figure 4 As shown, in the step S103, the step of converting the fused data set into a visual question answering data set includes steps S501 to S503:

[0065] S501: Based on the label set of the property data set , lists all object categories of interest ,in , W is the total number of categories;

[0066] S502: Based on the fusion data set Each image in and its corresponding object category , generate question-answer pairs ;

[0067] S503: Generate a visual question answering dataset according to the following formula: Where C represents the number of images screened.

[0068] In this example, the question You may ask what an object in the image is, or ask about some properties or relationships of the object, and the answer is a direct response to these questions. To construct a visual question answering dataset, this dataset consists of a series of question-answer pairs, each of which is combined with the fusion dataset is associated with an image in .

[0069] In one embodiment, if Figure 5As shown, in step S103, the step of integrating the professional knowledge of the property scene and constructing the visual-textual multimodal dataset includes steps S601 to S603:

[0070] S601. Professional knowledge in defining property scenarios , for each image and annotation , apply the defined conditional function Determine the corresponding decision information and output the Boolean value according to the following formula: ;

[0071] S602, if it exists If true, the corresponding question-answer pair is generated. ,in, represents the i-th question input, Indicates the answer to the corresponding question;

[0072] S603: Integrate all question-answer pairs into the visual question-answering dataset , build a visual-text multimodal dataset: .

[0073] In this embodiment, the following table shows some examples of property management scenario data. According to different business scenarios, corresponding professional knowledge is defined. For each image and annotation , apply the defined conditional function The corresponding decision information is judged to determine whether the annotation belongs to a specific scene attention label. If the judgment result is True, that is, there is a scene attention label, then the corresponding question-answer pair is generated. .in, represents the i-th question of the input, usually related to the object or scene in the image; Represents the answer to the corresponding question, that is, a direct response to the question. According to the definition of conditional function The judgment result is a Boolean value of True or False. True indicates that the label Expertise in real estate scenarios Related, that is, there is a scene attention label; False means the label Expertise in real estate scenarios Not relevant.

[0074] S104, fine-tuning the multimodal large model using a pre-trained visual model;

[0075] In this embodiment, in order to enable the multimodal large model to understand various business contexts, realize perception and decision-making of common scenarios, and enhance the practical application value of the model, a pre-trained visual model is used to fine-tune the multimodal large model.

[0076] In one embodiment, if Figure 6 As shown, in step S104, the step of fine-tuning the multimodal large model using the pre-trained visual model includes steps S701 to S703:

[0077] S701, using the pre-trained visual model to transform the image Convert to visual feature representation : ,in, Represents the parameters of the model VisionEncoder;

[0078] In this embodiment, the process is to convert the image This is done by inputting it into the VisionEncoder model and applying the model's parameters (represented by θ in the formula). Output visual feature representation is a vector or a collection of vectors that captures the key visual information in an image.

[0079] S702: Use a text feature extractor to extract the input question and obtain a text feature representation according to the following formula : ,in, Represents the parameters of the model TextEncoder;

[0080] In this embodiment, this process is similar to step S701, and the input (here is the question) is also This is done by inputting it into the TextEncoder model and applying the model's parameters (also represented by θ in the formula, but this may be the parameters of another model). Output text feature representation It is also a vector or a collection of vectors that captures the key textual information in the question.

[0081] S703, through the cross-modal attention mechanism, the visual features and the text features Fusion to get joint representation : ,in, Represents the parameters of the model Fusion.

[0082] In this embodiment, a cross-modal attention mechanism is used to represent visual features. and text features The representations are combined to obtain a joint representation This process aims to capture the correlation information between images and text and integrate this information into a unified representation. Joint Representation It can be used for subsequent tasks or decisions, such as image question answering, image description generation, etc.

[0083] S105. Deploy the fine-tuned multimodal large model into the property business system.

[0084] In this embodiment, the fine-tuned multimodal big model is integrated into the corresponding business system of the property to achieve online operation. By developing API interfaces or microservices, business applications can call the functions of the multimodal big model to achieve plug-and-play universal perception and intelligent decision-making in property scenarios.

[0085] In summary, if Figure 7 As shown in the figure, the overall process framework diagram of property scene perception and decision-making based on the visual-text multimodal large model is shown. Through label alignment and data screening mechanism based on image-level average precision sorting, the problems of data cross-domain and domain drift are solved. The integration of diverse data samples enhances the generalization ability of the model in different scenarios and environments, making it more robust in practical applications. The performance of the multimodal large model in the vertical field of property scenes is improved: the image data and professional knowledge from the perspective of the property scene monitoring camera are integrated, so that the multimodal large model has an in-depth understanding of the specific business background. The multimodal large model can more accurately identify and analyze key targets and events in the property scene, and enhance the perception and decision-making ability in the vertical field. Furthermore, in the fine-tuning process, only some parameters of the model are adjusted using LoRA technology, retaining the general representation ability of the pre-trained multimodal large model. The multimodal large model can handle general visual and language tasks, and adapt to specific business needs, realizing the effective combination of generality and professionalism. The specific steps are as described above and will not be described in detail here.

[0086] like Figure 8 As shown, an embodiment of the present invention also provides a property scene perception and decision-making device 800 based on a visual-textual multimodal large model, including an acquisition unit 801, a label alignment unit 802, a fusion unit 803, a fine-tuning unit 804, and a deployment unit 805.

[0087] An acquisition unit 801 is used to preprocess and annotate the target detection data set to a format specification to obtain a property data set containing multiple images;

[0088] The label alignment unit 802 is used to perform label alignment on the property data set and the public data set to obtain a common label set; and fuse the common label set through data screening to obtain a fused data set;

[0089] A fusion unit 803 is used to convert the fused data set into a visual question answering data set, and integrate the professional knowledge of the property scene to construct a visual-text multimodal data set, and use the visual-text multimodal data set to perform model training to obtain a multimodal large model;

[0090] A fine-tuning unit 804, configured to fine-tune the multimodal large model using a pre-trained visual model;

[0091] The deployment unit 805 is used to deploy the fine-tuned multimodal large model into the property business system.

[0092] The property scene perception and decision-making device based on the visual-textual multimodal large model provided by the embodiment of the present invention integrates visual data such as target detection data sets and public data sets with textual knowledge of business scenarios, fine-tunes the multimodal large model, so that it has the ability to understand contextual semantics and general scene perception, and is suitable for intelligent decision-making and perception in fields such as property management. First, in terms of property data set construction, the existing property scene (such as overflowing trash cans, illegal parking of motor vehicles, people falling, etc.) target detection data sets are used to integrate more visual public data sets to enhance the diversity of data samples and ensure data quality and model generalization capabilities. The labels of the target detection and instance segmentation data sets are sorted out to generate visual question and answer data in a format that supports instruction fine-tuning, providing training materials for the multimodal large model. Secondly, in terms of fine-tuning the multimodal large model, the visual question and answer data and the professional knowledge (know-how mechanism) of the property scene are integrated, and the pre-trained visual model is used for fine-tuning, so that the model can understand each business background, realize the perception and decision-making of general scenes, and enhance the practical application value of the multimodal large model.

[0093] The embodiment of the present invention also provides a property scene perception and decision-making device based on a visual-textual multimodal large model, which can be implemented in the form of a computer program. The computer program can be used in Fig. 9 Runs on the computer device shown.

[0094] See also Fig. 9 , Fig. 9 900 is a schematic block diagram of a computer device provided by an embodiment of the present invention. The computer device 900 is a server, which can be an independent server or a server cluster composed of multiple servers.

[0095] See also Fig. 9 The computer device 900 includes a processor 902 , a memory and a network interface 905 connected via a system bus 901 , wherein the memory may include a non-volatile storage medium 903 and an internal memory 904 .

[0096] The non-volatile storage medium 903 can store an operating system 9031 and a computer program 9032. When the computer program 9032 is executed, the processor 902 can execute a property scene perception and decision-making method based on a visual-textual multimodal large model.

[0097] The processor 902 is used to provide computing and control capabilities to support the operation of the entire computer device 900 .

[0098] The internal memory 904 provides an environment for the operation of the computer program 9032 in the non-volatile storage medium 903. When the computer program 9032 is executed by the processor 902, the processor 902 can execute a property scene perception and decision-making method based on a visual-textual multimodal large model.

[0099] The network interface 905 is used for network communication, such as providing data information transmission, etc. Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device 900 to which the solution of the present invention is applied. The specific computer device 900 may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0100] Those skilled in the art will understand that Fig. 9 The embodiments of the computer device shown in the figure do not constitute a limitation on the specific composition of the computer device. In other embodiments, the computer device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. For example, in some embodiments, the computer device may only include a memory and a processor. In such embodiments, the structure and function of the memory and the processor are the same as those of the embodiment of the present invention. Fig. 9 The embodiments shown are consistent and will not be described again here.

[0101] It should be understood that in the embodiment of the present invention, the processor 902 may be a central processing unit (CPU), and the processor 902 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0102] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the above-described equipment, devices and units can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. Those of ordinary skill in the art can appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, computer software or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to the function. Whether these functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0103] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, and units with the same function may be combined into one unit, for example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0104] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present invention.

[0105] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0106] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), disk or optical disk and other media that can store program code.

[0107] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A property scene perception and decision-making method based on a visual-textual multimodal large model, characterized in that: include: Preprocess and annotate the target detection dataset to obtain a property dataset containing multiple images; Aligning labels of the property dataset with a public dataset to obtain a common label set; and fusing the common label set through data screening to obtain a fused data set; Converting the fused dataset into a visual question answering dataset, integrating professional knowledge of property scenarios, constructing a visual-text multimodal dataset, and using the visual-text multimodal dataset for model training to obtain a multimodal large model; Fine-tuning the multimodal large model using a pre-trained visual model; Deploy the fine-tuned multimodal large model into the property business system; The property dataset is labeled with the public dataset to obtain a common label set, including: The label set of the property dataset is defined as follows: The label set of the public dataset is defined as follows: According to the label set of the property dataset and the label set of the public dataset, a label mapping function is established according to the following formula to map the label of the public dataset to the label of the property dataset to obtain a valid property label: in, express The i-th label in , i∈[1,n], n is a natural number; express The jth label in , j∈[1,m], m is a natural number; "φ" means that the label of the public dataset is not mapped to the label of the property dataset; According to the valid property labels, the total label set is obtained according to the following formula: The label alignment of the property dataset and the public dataset to obtain a common label set also includes: The annotation of the public dataset is defined as follows: label(b)∈L pu ,b∈B pu ; According to the label mapping function, the annotations of the public dataset are converted into property labels according to the following formula: label mapped (b)=f(label(b)), if label mapped (b) = φ, then delete the annotation; The method of fusing the common label set through data screening to obtain a fused data set includes: Through the property dataset D pr Train to get an original model M pr ; By the original model M pr The images in the public dataset are inferred and the prediction results are obtained according to the following formula: i =M pr (I i |θ * ), where I i is the i-th image, θ * are the parameters of the trained original model; The real annotation of the public dataset Mapped to the label of the property dataset by the label mapping function: According to the real annotation of the public dataset Filter out the annotations of the common tag set: According to the prediction results and the annotation of the common label set The average precision AP(I i ): Among them, P interp represents the interpolation accuracy at the recall level r, R represents the set of recall values ​​from 0 to 1 with an increment of 0.1, and K represents the number of different recall points used for calculation in the recall rate R; All images are sorted according to the average precision AP(I i ) Sort from high to low to get a sorted list Among them, M means that there are M average precision AP (I i ), the filtered fusion data set D is obtained according to the following formula f :D f = {I i |i=1,2,…,C}, where C represents the number of images screened out.

2. The property scene perception and decision-making method based on the visual-textual multimodal large model according to claim 1 is characterized in that: The converting the fused dataset into a visual question answering dataset comprises: According to the label set L of the property dataset pr , list all object categories of interest l w , where w = 1, 2, 3, …, W, W is the total number of categories; According to the fusion dataset D f Each image in and its corresponding object category l w , generate question-answer pairs (Q i,w ,A i,w ); Generate the visual question answering dataset as follows: Where C represents the number of images screened.

3. The property scene perception and decision-making method based on the visual-textual multimodal large model according to claim 2 is characterized in that: The above-mentioned expertise in real estate scenarios is incorporated into the construction of a visual-textual multimodal dataset, including: Expertise in defining property scenarios S , for each image I i and marked B i , apply the defined conditional function C S (I i ,B i ) determines the corresponding decision information and outputs a Boolean value according to the following formula: If there is C S (I i ,B i ) is true, then the corresponding question-answer pair (Q i ,A i ), where Q i represents the i-th question input, A i Indicates the answer to the corresponding question; Incorporate all question-answer pairs into the visual question answering dataset Constructing a visual-textual multimodal dataset:

4. The property scene perception and decision-making method based on the visual-textual multimodal large model according to claim 3 is characterized in that: The method of fine-tuning the multimodal large model using a pre-trained visual model includes: Use the pre-trained visual model to transform the image I into i Converted into visual feature representation V i : V i =VisionEncoder(I i |θ v ), where θ v Represents the parameters of the VisionEncoder model; Use the text feature extractor to extract the input question and get the text feature representation T according to the following formula i :T i =TextEncoder(Q i |θ t ), where θ t Represents the parameters of the TextEncoder model; Through the cross-modal attention mechanism, the visual feature V i and the text feature T i Fusion, get the joint representation H i : H i =Fusion(V i ,T i |θ f ), where θ f Represents the parameters of the Fusion function.

5. A property scene perception and decision-making device based on a visual-textual multimodal large model, characterized in that: include: An acquisition unit, used for preprocessing and annotating the target detection data set to a format specification, and obtaining a property data set containing multiple images; A label alignment unit, used to perform label alignment on the property dataset and the public dataset to obtain a common label set; and fusing the common label set through data screening to obtain a fused data set; A fusion unit, used to convert the fused data set into a visual question answering data set, integrate the professional knowledge of the property scene, construct a visual-text multimodal data set, and use the visual-text multimodal data set to perform model training to obtain a multimodal large model; A fine-tuning unit, used to fine-tune the multimodal large model using a pre-trained visual model; A deployment unit, used to deploy the fine-tuned multimodal large model into the property business system; The property dataset is label-aligned with the public dataset to obtain a common label set, including: The label set of the property dataset is defined as follows: The label set of the public dataset is defined as follows: According to the label set of the property dataset and the label set of the public dataset, a label mapping function is established according to the following formula to map the label of the public dataset to the label of the property dataset to obtain a valid property label: in, express The i-th label in , i∈[1,n], n is a natural number; express The jth label in , j∈[1,m], m is a natural number; "φ" means that the label of the public dataset is not mapped to the label of the property dataset; According to the valid property labels, the total label set is obtained according to the following formula: The label alignment of the property dataset and the public dataset to obtain a common label set also includes: The annotation of the public dataset is defined as follows: label(b)∈L pu ,b∈B pu ; According to the label mapping function, the annotations of the public dataset are converted into property labels according to the following formula: label mapped (b)=f(label(b)), if label mapped (b) = φ, then delete the annotation; The method of fusing the common label set through data screening to obtain a fused data set includes: Through the property dataset D pr Train to get an original model M pr ; By the original model M pr The images in the public dataset are inferred and the prediction results are obtained according to the following formula: i =M pr (I i |θ * ), where I i is the i-th image, θ * are the parameters of the trained original model; The real annotation of the public dataset Mapped to the label of the property dataset by the label mapping function: According to the real annotation of the public dataset Filter out the annotations of the common tag set: According to the prediction results and the annotation of the common label set The average precision AP(I i ): Among them, P interp represents the interpolation accuracy at the recall level r, R represents the set of recall values ​​from 0 to 1 with an increment of 0.1, and K represents the number of different recall points used for calculation in the recall rate R; All images are sorted according to the average precision AP(I i ) Sort from high to low to get a sorted list Among them, M means that there are M average precision AP (I i ), the filtered fusion data set D is obtained according to the following formula f :D f = {I i |i=1,2,…,C}, where C represents the number of images screened out.

6. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it implements the property scene perception and decision-making method based on the visual-textual multimodal large model as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, enables the processor to execute the property scene perception and decision-making method based on a visual-textual multimodal large model as described in any one of claims 1 to 4.

Images