Business object state evaluation method and apparatus, electronic device, and readable storage medium

Abstract

This disclosure relates to the field of object evaluation technology, and provides a method, apparatus, electronic device, and readable storage medium for evaluating the state of a business object. The method includes: performing visual structuring processing on a visual image to be processed to obtain a target region mask image; performing feature extraction processing on environmental audio data and business data to be processed to obtain audio feature vectors and structured keyword data; performing multimodal recognition processing on the target region mask image, audio feature vectors, and structured keyword data respectively to obtain visual feature vectors, audio feature vectors, and text feature vectors; performing cross-modal attention fusion processing on the above feature vectors to obtain fused feature vectors; and performing gradient boosting processing on the fused feature vectors to obtain the target business object state evaluation result. This improves accuracy, enhances understanding, increases processing efficiency and robustness of the evaluation results, improves adaptability, and enhances dynamic iteration capabilities.

Description

Technical Field

[0001] This disclosure relates to the field of object assessment technology, and in particular to a method, apparatus, electronic device, and readable storage medium for assessing the status of business objects. Background Technology

[0002] Currently, asset value determination mainly relies on visual recognition methods, such as using fixed cameras to collect images and comparing them with historical feature databases using convolutional neural network (CNN) models to identify illegal buildings. However, this method only focuses on the appearance changes of completed buildings and cannot cover the early stages of illegal activities, such as material transportation and construction processes, resulting in serious delays in early warnings. At the same time, visual recognition is easily affected by environmental interference in complex scenarios such as tree obstruction, backlighting, or nighttime environments, leading to a high false judgment rate and low accuracy of early warnings.

[0003] It is evident that existing technologies suffer from low accuracy in early warning, poor dynamic iteration capability, and poor adaptability due to the inability to collaboratively process multi-source data. Summary of the Invention

[0004] In view of this, the present disclosure provides a business object status assessment method, apparatus, electronic device and readable storage medium to solve the problems of low early warning accuracy, poor dynamic iteration capability and poor adaptability in the prior art due to the inability to collaboratively process multi-source data.

[0005] A first aspect of this disclosure provides a method for evaluating the state of a business object, comprising: performing visual structuring processing on a visual image to be processed to obtain a target region mask image; performing feature extraction processing on environmental audio data and business data to be processed to obtain audio feature vectors and structured keyword data; performing multimodal recognition processing on the target region mask image, audio feature vectors, and structured keyword data respectively to obtain visual feature vectors, audio feature vectors, and text feature vectors; performing cross-modal attention fusion processing on the visual feature vectors, audio feature vectors, and text feature vectors to obtain fused feature vectors; and performing gradient boosting processing on the fused feature vectors to obtain a target business object state evaluation result.

[0006] In some embodiments, gradient boosting is performed on the fused feature vector to obtain the state evaluation result of the target business object, including: performing confidence calculation on the fused feature vector to obtain a confidence label; performing weight adaptive processing on the fused feature vector based on the confidence label to obtain a fused feature correction vector; and performing iterative evaluation processing on the fused feature correction vector to obtain the state evaluation result of the target business object.

[0007] In some embodiments, cross-modal attention fusion processing is performed on visual feature vectors, audio feature vectors, and text feature vectors to obtain fused feature vectors, including: aligning the visual feature vectors, audio feature vectors, and text feature vectors according to their channel dimensions to obtain a channel feature set; performing cross-modal correlation calculation processing on the channel feature set to obtain a correlation weight matrix; and performing weighted fusion processing on the channel feature set using the correlation weight matrix to obtain the fused feature vector.

[0008] In some embodiments, performing visual structuring processing on the visual image to be processed to obtain a target region mask image includes: performing illumination equalization processing on the visual image to be processed to obtain an illumination compensation image; performing background region segmentation processing on the illumination compensation image to obtain an initial mask image; and performing edge fidelity filtering processing on the initial mask image to obtain a target region mask image.

[0009] In some embodiments, the environmental audio data is subjected to feature extraction processing to obtain an audio feature vector, including: performing steady-state noise suppression processing on the environmental audio data to obtain noise-reduced frequency data; performing joint feature extraction processing on the noise-reduced frequency data to obtain the spectral features corresponding to the environmental audio data; and performing principal component analysis processing on the spectral features to obtain the audio feature vector.

[0010] In some embodiments, feature extraction processing is performed on the business data to be processed to obtain structured keyword data, including: normalizing the fields of the business data to be processed to obtain standardized business data; calculating keyword weights on the standardized business data to obtain a keyword weight sequence; and filtering the keyword weight sequence based on a preset weight threshold to obtain structured keyword data.

[0011] In some embodiments, the channel feature set is weighted and fused using an association weight matrix to obtain a fused feature vector, including: weighting the channel feature set based on the association weight matrix to obtain a weighted feature subset; performing information filtering on the weighted feature subset to obtain a compressed feature set; and performing feature concatenation on the compressed feature set to obtain a fused feature vector.

[0012] A second aspect of this disclosure provides a business object state assessment apparatus, comprising: a first processing module for performing visual structuring processing on a visual image to be processed to obtain a target region mask image; a second processing module for performing feature extraction processing on environmental audio data and business data to be processed to obtain audio feature vectors and structured keyword data; a third processing module for performing multimodal recognition processing on the target region mask image, audio feature vectors, and structured keyword data respectively to obtain visual feature vectors, audio feature vectors, and text feature vectors; a fourth processing module for performing cross-modal attention fusion processing on the visual feature vectors, audio feature vectors, and text feature vectors to obtain fused feature vectors; and a fifth processing module for performing gradient boosting processing on the fused feature vectors to obtain a target business object state assessment result.

[0013] A third aspect of this disclosure provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described above.

[0014] A fourth aspect of this disclosure provides a readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method.

[0015] The beneficial effects of this embodiment compared with the prior art are as follows: Visual structuring is performed on the visual image to be processed to obtain a target region mask image; feature extraction is performed on the environmental audio data and the business data to be processed to obtain audio feature vectors and structured keyword data; multimodal recognition processing is performed on the target region mask image, audio feature vectors, and structured keyword data respectively to obtain visual feature vectors, audio feature vectors, and text feature vectors; cross-modal attention fusion processing is performed on the visual feature vectors, audio feature vectors, and text feature vectors to obtain fused feature vectors; gradient boosting processing is performed on the fused feature vectors to obtain the target business object state evaluation result. This improves the accuracy of business object state evaluation, enhances the understanding of complex scenarios, improves the efficiency of multimodal data fusion and the robustness of the evaluation results, enhances adaptability, and strengthens dynamic iteration capabilities. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram illustrating an application scenario of an embodiment of this disclosure; Figure 2 This is a flowchart illustrating a business object status assessment method provided in an embodiment of this disclosure; Figure 3 This is a flowchart illustrating another business object status assessment method provided in this embodiment of the disclosure; Figure 4 This is a schematic flowchart of a multimodal data acquisition and preprocessing method provided in an embodiment of this disclosure; Figure 5 This is a flowchart illustrating a multimodal artificial intelligence recognition method provided in an embodiment of this disclosure; Figure 6 This is a flowchart illustrating a multimodal fusion decision-making and closed-loop iterative method provided in an embodiment of this disclosure; Figure 7 This is a schematic diagram of the structure of a business object status assessment device provided in an embodiment of this disclosure; Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0018] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, so as to provide a thorough understanding of the embodiments of this disclosure. However, those skilled in the art will understand that this disclosure may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this disclosure with unnecessary detail.

[0019] It should be noted that the user information (including but not limited to terminal device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0020] A method and apparatus for evaluating the status of a business object according to an embodiment of the present disclosure will now be described in detail with reference to the accompanying drawings.

[0021] Figure 1 This is a schematic diagram illustrating an application scenario of an embodiment of this disclosure. The application scenario may include terminal devices 1, 2, and 3, server 4, and network 5.

[0022] Terminal devices 1, 2, and 3 can be hardware or software. When terminal devices 1, 2, and 3 are hardware, they can be various electronic devices with displays and supporting communication with server 4, including but not limited to smartphones, tablets, laptops, and desktop computers. When terminal devices 1, 2, and 3 are software, they can be installed in the aforementioned electronic devices. Terminal devices 1, 2, and 3 can be implemented as multiple software programs or software modules, or as a single software program or software module; this disclosure does not limit this. Furthermore, various applications can be installed on terminal devices 1, 2, and 3, such as data processing applications, instant messaging tools, social platform software, search applications, shopping applications, etc.

[0023] Server 4 can be a server that provides various services, such as a backend server that receives requests sent by terminal devices with which it has established communication connections. This backend server can receive and analyze the requests sent by the terminal devices and generate processing results. Server 4 can be a single server, a server cluster consisting of several servers, or a cloud computing service center. This disclosure embodiment does not limit this.

[0024] It should be noted that server 4 can be either hardware or software. When server 4 is hardware, it can be various electronic devices that provide various services to terminal devices 1, 2, and 3. When server 4 is software, it can be multiple software programs or software modules that provide various services to terminal devices 1, 2, and 3, or it can be a single software program or software module that provides various services to terminal devices 1, 2, and 3. This disclosure does not limit the scope of the embodiments.

[0025] Network 5 can be a wired network using coaxial cable, twisted pair, and fiber optic connection, or it can be a wireless network that enables interconnection of various communication devices without wiring, such as Bluetooth, Near Field Communication (NFC), and Infrared. This disclosure does not limit the scope of the network.

[0026] Users can establish a communication connection with server 4 via network 5 through terminal devices 1, 2, and 3 to receive or send information. Specifically, server 4 can acquire visual images to be processed, environmental audio data, and business data to be processed through terminal devices 1, 2, and 3. Visual structuring processing is performed on the visual images to be processed to obtain a target region mask image. Feature extraction processing is performed on the environmental audio data and the business data to be processed to obtain audio feature vectors and structured keyword data. Multimodal recognition processing is performed on the target region mask image, audio feature vectors, and structured keyword data to obtain visual feature vectors, audio feature vectors, and text feature vectors. Cross-modal attention fusion processing is performed on the visual feature vectors, audio feature vectors, and text feature vectors to obtain a fused feature vector. Gradient boosting processing is performed on the fused feature vector to obtain the target business object state evaluation result.

[0027] It should be noted that the specific types, quantities, and combinations of terminal devices 1, 2, and 3, server 4, and network 5 can be adjusted according to the actual needs of the application scenario, and this disclosure embodiment does not impose any restrictions on this.

[0028] Figure 2 This is a flowchart illustrating a business object status assessment method provided in an embodiment of this disclosure. Figure 2 The business object status assessment method can be derived from Figure 1 The server executes this. For example... Figure 2 As shown, the business object status assessment method includes: S201, Perform visual structuring processing on the visual image to be processed to obtain a mask image of the target region.

[0029] Specifically, the visual image to be processed can be raw image data of the building exterior collected by fixed cameras or drones deployed at key locations in the property community. The form of the visual image to be processed can be a digital representation of visual information including the building facade, public areas, or aerial views, thus providing an initial input data source for subsequent business object status assessment. The visual image to be processed can be acquired by capturing and transmitting it to the processing system through image acquisition equipment. Visual structured processing can be a pixel-level classification process of the input visual image to be processed using a semantic segmentation model. This visual structured processing can divide the visual image to be processed into different semantic regions through deep learning algorithms, thereby separating the main structure of the building from the background. Among them, the target region mask image can be a binary image generated by semantic segmentation. The form of the target region mask image can be matrix data that distinguishes the target building area from the background by pixel values, thereby identifying the potential areas of the visual image to be evaluated and providing a basis for subsequent differential comparison.

[0030] Furthermore, the semantic segmentation model can be a fully dimensional, densely nested image segmentation architecture (U-Net++). This semantic segmentation model can extract multi-scale features through an encoder-decoder structure and fuse deep and shallow features through skip connections to improve edge segmentation accuracy. It can also perform image enhancement processing on the visual image to be processed, optimize uneven lighting scenes through the Retinex algorithm, and then input the semantic segmentation model to generate a mask image of the target region.

[0031] For example, in an application scenario based on monitoring illegal construction in a residential community, a fixed camera can be used to collect a visual image of the facade of a building to be processed. This visual image may contain tree occlusion and backlight interference. A pre-trained U-Net++ model can be loaded through visual structured processing to perform semantic segmentation on the input visual image. The encoder can extract the building outline features, and the decoder can gradually upsample to restore spatial details, resulting in a target area mask image. This target area mask image can use white pixels to identify the main building area and black pixels to represent the background.

[0032] This application embodiment transforms the visual image to be processed into a target region mask image through visual structuring processing, accurately extracts building regions through semantic segmentation, and combines image enhancement to optimize adaptability to complex scenes. This improves the accuracy and robustness of subsequent business object state assessment. By eliminating environmental interference factors through the target region mask image, it ensures that differential comparison focuses only on the target region, reduces the false judgment rate, and enhances stability under conditions such as occlusion and backlighting.

[0033] S202, the environmental audio data and the business data to be processed are subjected to feature extraction processing to obtain audio feature vectors and structured keyword data.

[0034] Specifically, environmental audio data can be physical environmental sound signals collected by microphones deployed within the property community, including audio inputs from building unit entrances and public areas. This environmental audio data can be in the form of sound waveform information of the monitored area and can be used to acquire noise events caused by illegal construction or material transportation, etc., without limitation here. Business data to be processed can be structured or semi-structured records obtained from the property management system through an interface, including work order data such as garbage collection, maintenance, and complaints, as well as visitor access data such as identity information, facial images, access time, and vehicle information, without limitation here. This business data to be processed can also be business logs and transaction data generated during property operation, which can be used to record personnel activities and resource flow. Feature extraction processing can be the process of transforming environmental audio data and business data to be processed to extract representative features. This can transform environmental audio data and business data to be processed into numerical forms suitable for input to artificial intelligence (AI) models, facilitating subsequent multimodal fusion. For example, Mel-Frequency Cepstral Coefficients (MFCCs) can be used to process environmental audio data, and Term Frequency-Inverse Document Frequency (Term Frequency-Inverse Document Frequency) can be used. Frequency-Inverse Document Frequency (TF-IDF) can be used to process business data to be processed.

[0035] The audio feature vector can be a numerical sequence extracted from environmental audio data. This audio feature vector can contain time-domain and frequency-domain features of the audio, such as Mel-frequency cepstral coefficients and short-time Fourier transform coefficients (STFT). The form of the audio feature vector can be a mathematical representation of the audio signal, which can be used to quantify sound features. The structured keyword data can be a set of semantic units extracted from the business data to be processed, such as "construction waste" or "wall demolition," etc. There is no limitation here. The importance can be represented by TF-IDF weighting. The form of the structured keyword data can be a list of key terms in text data, which can be used to summarize business content.

[0036] Furthermore, in feature extraction processing, environmental audio data can be processed to remove environmental interference through wavelet transform, and features can be extracted using MFCC and STFT algorithms to form audio feature vectors, which can be used for noise classification processing in subsequent CNN and Long Short-Term Memory Network (LSTM) models. Business data to be processed can be processed using TF-IDF to extract high-frequency keywords from work order texts to obtain structured keyword data, which can be used for risk assessment of Light Gradient Boosting Machine (LightGBM) models.

[0037] For example, in the scenario of monitoring illegal construction in a residential community, environmental audio data, such as construction machinery noise, can be continuously collected through microphones in the community. Business data to be processed, such as frequent construction waste removal work orders, can be provided through the property management system. Through feature extraction processing, environmental audio data can be converted into audio feature vectors, and business data to be processed can be converted into structured keyword data.

[0038] This application embodiment transforms environmental audio data and business data to be processed into audio feature vectors and structured keyword data through feature extraction processing. By integrating audio and business features, it improves the coverage and accuracy of business object status assessment. Feature extraction of environmental audio data expands the monitoring dimensions, and the generation of structured keyword data enhances the semantic understanding capability of business data to be processed. This improves the timeliness of early warning, reduces misjudgments caused by data isolation, and supports full-cycle business object status assessment.

[0039] S203, perform multimodal recognition processing on the target region mask image, audio feature vector and structured keyword data respectively to obtain visual feature vector, audio feature vector and text feature vector.

[0040] Specifically, multimodal recognition processing can include the process of extracting and transforming features from corresponding input data using visual recognition models, audio recognition models, and text recognition models, respectively. The visual feature vector can be a high-dimensional numerical representation extracted from the target region mask image by the visual recognition model, which can be used to characterize the visual changes of business objects in subsequent fusion decisions. This visual feature vector can be obtained by feature encoding the target region mask image using a Vision Transformer (ViT) model or a single-stage object detection (YOLOv9) model. The audio feature vector can be an enhanced feature representation obtained by processing the input environmental audio data using an audio recognition model. The text feature vector can be a semantic embedding representation extracted from structured keyword data using a text recognition model.

[0041] Furthermore, visual recognition processing can use the ViT model to calculate the structural difference between the target region mask image and the benchmark feature library, and output a visual feature vector; audio recognition processing can use CNN and LSTM models to classify illegal noise in audio feature vectors and output an enhanced audio feature vector; text recognition processing can use the Bidirectional Encoder Representations from Transformer (BERT) model to semantically encode structured keyword data and obtain text feature vectors.

[0042] In addition, multimodal recognition processing can utilize a differential comparison mechanism, using the ViT model to extract the current structural features of the target area mask image and calculate similarity with standard building features in a benchmark feature library. The benchmark feature library can contain feature representations of the original design drawings of the community and compliant handover images, which can be used to provide a reference benchmark for visual comparison. Audio recognition processing can utilize a temporal modeling mechanism, using CNN and LSTM models to analyze the time series patterns of audio feature vectors and identify the persistence and intensity characteristics of illegal construction noise. Illegal construction noise can be audio events with specific frequencies and energy distributions, which can be used to associate with construction behavior. Text recognition processing can utilize a semantic analysis mechanism, using the BERT model to parse the contextual associations of structured keyword data and output the potential risk features of the work order content. Work order risk features can be text patterns used to characterize early warning signs, which can be used for early warning priority determination.

[0043] This application embodiment generates visual, audio, and text feature vectors by performing multimodal recognition processing on the target region mask image, audio feature vector, and structured keyword data respectively, realizing independent feature extraction and standardized representation of multi-source data. By processing different types of data through a dedicated model for each modality, the accuracy and consistency of features of each modality are ensured, improving the extraction accuracy of visual, audio, and text features, providing reliable input for subsequent multimodal fusion decision-making, and enhancing the accuracy and timeliness of business object status assessment and early warning.

[0044] For example, in the scenario of monitoring illegal construction in residential communities, a target area mask image can be generated using the U-Net++ model to focus on potentially illegal building areas; an audio feature vector can be generated using wavelet transform and MFCC feature extraction to characterize construction noise; structured keyword data containing information such as the frequency of construction waste removal can be generated using TF-IDF processing; then, visual features can be extracted from the target area mask image using the ViT model, and the output visual feature vector can be used to characterize changes in building structure; audio feature vectors can be enhanced using CNN and LSTM models, and the output audio feature vector can be used to characterize the probability of noise events; and text features can be extracted from the structured keyword data using the BERT model, and the output text feature vector can be used to characterize the risk level of work orders.

[0045] S204 performs cross-modal attention fusion processing on the visual feature vector, audio feature vector, and text feature vector to obtain the fused feature vector.

[0046] Specifically, cross-modal attention fusion processing can be a multimodal data integration process that can calculate the correlation weights between features of different modalities through attention mechanisms to achieve feature alignment and enhancement. This cross-modal attention fusion processing can be performed by a cross-modal attention fusion network, which assigns the contribution of visual, audio and text features through multiple attention heads.

[0047] The fusion feature vector can be a unified numerical representation generated after cross-modal attention fusion processing. This fusion feature vector can be used to characterize multi-dimensional information about the risk of illegal buildings.

[0048] Furthermore, the cross-modal attention fusion network can normalize the visual feature vector, audio feature vector, and text feature vector to eliminate dimensional differences; it can determine the weight ratio of each modality feature in the fusion process through attention score calculation; and then it can concatenate the weighted features into a high-dimensional vector to output the fused feature vector.

[0049] For example, in the scenario of identifying illegal buildings in a residential community, visual feature vectors can be generated by collecting building images through fixed cameras, audio feature vectors can be generated by collecting environmental audio through microphones, and text feature vectors can be generated by obtaining work order records through the property management system interface. The above vectors can be input into a cross-modal attention fusion network. The cross-modal attention fusion network can focus on changes in building outlines in visual data, peak construction noise in audio data, and high-frequency cleaning work orders in text data through attention mechanisms to generate fused feature vectors.

[0050] In addition, cross-modal attention fusion processing can also include feature redundancy elimination and dimensionality compression. Visual feature vectors may contain repetitive building outline information, audio feature vectors may contain irrelevant environmental noise, and text feature vectors may contain low-risk work order data. Through cross-modal attention fusion processing, highly relevant feature components can be retained and redundant parts can be eliminated through attention filtering.

[0051] This application embodiment integrates visual, audio, and text feature vectors through cross-modal attention fusion processing, realizing collaborative evaluation of multi-source data and avoiding misjudgments caused by isolated analysis of single-modal data; by dynamically weighting different modal features through an attention mechanism, the accuracy and robustness of feature fusion are improved, the accuracy of early warning is enhanced, and invalid checks are reduced.

[0052] S205, perform gradient boosting on the fused feature vector to obtain the target business object status evaluation result.

[0053] Specifically, gradient boosting can be performed on the fused feature vectors using the extreme gradient boosting (XGBoost) model. The XGBoost model can optimize prediction accuracy by iteratively constructing multiple decision trees. Each decision tree can be trained based on the residual of the previous decision tree. The outputs of all decision trees are integrated to obtain a comprehensive score. Gradient boosting can be used to map multimodal features to state assessment levels or violation risk levels. The state assessment result of the target business object can be the violation risk level classification output by the gradient boosting process, such as including three levels: high risk, medium risk, and low risk, etc., without limitation here. The state assessment result of the target business object can be used to indicate the probability of illegal buildings in the property community and can provide a decision basis for early warning push.

[0054] Furthermore, gradient boosting can standardize the fused feature vectors to eliminate the influence of dimensions, and then input them into the XGBoost model for forward propagation calculation. Each decision tree node can be split based on the components of the feature vector, and the target business object status evaluation result is output through a weighted voting mechanism.

[0055] This application embodiment achieves the classification of business object status by performing gradient boosting processing on the fused feature vector and integrating multimodal features through the XGBoost model. By combining multi-source information fusion with machine learning algorithms, the accuracy and reliability of early warning decisions are improved, and invalid checks caused by misjudgment due to single data are avoided. The iterative optimization mechanism of gradient boosting processing enhances the adaptive capability, ensuring that the evaluation results are dynamically adjusted according to the usage scenario, and improving the overall efficiency of business object status evaluation.

[0056] For example, in the scenario of monitoring illegal construction in residential communities, the fused feature vector can include multiple dimensions such as visual difference, material confidence, noise confidence, duration, work order risk level, and personnel matching. The fused feature vector can be subjected to gradient boosting, and the fused feature vector can be standardized and classified through the XGBoost model to obtain the status assessment result of the target business object, that is, the illegal risk level. For example, when the fused feature vector represents high building difference, high material confidence, and is accompanied by long-term illegal noise, a high risk level can be output.

[0057] According to the technical solution provided in this disclosure, a target region mask image is obtained by performing visual structuring processing on the visual image to be processed; feature extraction processing is performed on the environmental audio data and the business data to be processed to obtain audio feature vectors and structured keyword data; multimodal recognition processing is performed on the target region mask image, audio feature vectors, and structured keyword data respectively to obtain visual feature vectors, audio feature vectors, and text feature vectors; cross-modal attention fusion processing is performed on the visual feature vectors, audio feature vectors, and text feature vectors to obtain fused feature vectors; gradient boosting processing is performed on the fused feature vectors to obtain the target business object state evaluation result, thereby improving the accuracy of business object state evaluation, enhancing the understanding of complex scenarios, improving the efficiency of multimodal data fusion and the robustness of evaluation results, improving adaptability, and enhancing dynamic iteration capability.

[0058] In some embodiments, gradient boosting is performed on the fused feature vector to obtain the state evaluation result of the target business object, including: performing confidence calculation on the fused feature vector to obtain a confidence label; performing weight adaptive processing on the fused feature vector based on the confidence label to obtain a fused feature correction vector; and performing iterative evaluation processing on the fused feature correction vector to obtain the state evaluation result of the target business object.

[0059] Specifically, confidence calculation can be a probability-statistic-based evaluation process. Confidence calculation can quantify the reliability of prediction results by calculating the distribution stability of the probability output by the model, thereby providing an adjustment benchmark for weight adaptive processing. The confidence label can be a quantitative indicator obtained from the confidence calculation, and the confidence label can be in the form of a numerical label, which can be used to characterize the reliability level of the evaluation results. This can serve as a reference factor for adjusting feature weights in weight adaptive processing. The confidence label can be generated by statistical analysis of the fused feature vector through confidence calculation.

[0060] Furthermore, the confidence score calculation can be performed by analyzing the variance of the probability distribution output by the XGBoost model and combining it with historical early warning feedback data. This allows for the calculation of the confidence score for each fused feature vector and the generation of the corresponding confidence label.

[0061] In addition, weight adaptive processing can be a process of dynamically adjusting feature weights. Different weights can be assigned to each dimension of the fused feature vector according to the confidence label, thereby optimizing the feature representation to reduce the impact of low-confidence features. The fused feature correction vector can be the feature representation obtained by weight adaptive processing. The form of the fused feature correction vector can be a multimodal feature vector obtained by weight adjustment processing, thereby providing more stable input data for iterative evaluation processing.

[0062] Furthermore, the weight adaptive processing can be achieved through linear weighting, dynamically adjusting the weight ratio of visual, audio, and business features in the fused feature vector according to the confidence level of the labels. For example, the weight of visual features can be reduced for low confidence labels to cope with interference in complex scenes. This weight adaptive processing enhances the robustness to noisy data through feature weight optimization.

[0063] Furthermore, iterative evaluation processing can be a process of gradually optimizing the evaluation results. Iterative evaluation processing can refine the fused feature correction vector by repeating the evaluation process multiple times, thereby improving the accuracy and stability of the evaluation results.

[0064] Furthermore, the iterative evaluation process can be performed through multiple rounds of prediction using the XGBoost model. In each round, the model parameters are adjusted based on the residuals of the previous round, and the fused feature correction vector is repeatedly evaluated until a stable target business object status evaluation result is output. This iterative evaluation process effectively integrates key information from multimodal features through iterative optimization, thereby improving the accuracy of business object status evaluation.

[0065] For example, in the identification of illegal buildings in residential communities, a confidence label can be calculated on the fused feature vector. Based on the confidence label, the weights of the fused feature vector can be adaptively adjusted to obtain a fused feature correction vector. Then, the status evaluation result of the target business object can be output through iterative evaluation.

[0066] According to the technical solution provided in this disclosure, a confidence label is generated by calculating the confidence level of the fused feature vector. Based on the confidence label, the fused feature vector is subjected to adaptive weight processing to obtain a fused feature correction vector. Then, the fused feature correction vector can be iteratively evaluated to output the target business object status evaluation result. Through confidence-driven weight adjustment and iterative optimization, the stability and evaluation accuracy of multimodal feature fusion are improved, the misjudgment rate in complex scenarios is reduced, and the reliability of business object status evaluation is enhanced.

[0067] In some embodiments, cross-modal attention fusion processing is performed on visual feature vectors, audio feature vectors, and text feature vectors to obtain fused feature vectors, including: aligning the visual feature vectors, audio feature vectors, and text feature vectors according to their channel dimensions to obtain a channel feature set; performing cross-modal correlation calculation processing on the channel feature set to obtain a correlation weight matrix; and performing weighted fusion processing on the channel feature set using the correlation weight matrix to obtain the fused feature vector.

[0068] Specifically, channel dimension alignment can be a data standardization process that maps feature vectors from different modalities to a unified dimensional space. This channel dimension alignment can be used to eliminate feature dimension differences to achieve cross-modal interaction. The channel feature set can be a combination of multimodal features formed through dimension alignment, which can be used as the input basis for cross-modal attention fusion.

[0069] For example, in the scenario of identifying illegal buildings in residential communities, channel dimension alignment can be achieved by projecting the 512-dimensional visual feature vector, the 128-dimensional audio feature vector, and the 256-dimensional text feature vector into a 1024-dimensional space through a fully connected layer, and eliminating distribution bias through layer normalization.

[0070] In addition, the cross-modal correlation calculation process can be a process of calculating the correlation between different modal features through an attention mechanism, which can be used to determine the contribution of each modal feature in the fusion process; the correlation weight matrix can be a numerical matrix used to characterize the interaction strength between multimodal features, and the correlation weight matrix can be used to guide feature selection during weighted fusion.

[0071] For example, in the scenario of identifying illegal buildings in residential communities, cross-modal correlation calculation can be processed through a 4-head attention mechanism, where the correlation between visual features and audio features can be obtained by calculating cosine similarity, which is used to evaluate the spatiotemporal consistency between construction noise and building changes.

[0072] Furthermore, the process of generating the association weight matrix may also include splitting the channel feature set into query vectors, key vectors, and value vectors, and calculating the interaction weights between modalities through scaling dot product attention.

[0073] Furthermore, weighted fusion processing can be a data integration process that linearly combines multimodal features based on the correlation weight matrix, and can be used to generate a unified feature representation with cross-modal information.

[0074] For example, in the scenario of identifying illegal buildings in residential communities, a 1024-dimensional fused feature vector can be generated by weighted fusion processing, which combines the building difference of the visual feature vector, the duration of the audio feature vector, and the work order risk level of the text feature vector according to the association weight matrix.

[0075] Furthermore, weighted fusion processing can be achieved through an element-wise weighting process. When the weights of visual and textual modalities are higher in the correlation weight matrix, it can indicate that scenarios with strong correlation between work order data and image changes have higher decision priority.

[0076] According to the technical solution provided in this disclosure, by aligning the visual, audio and text feature vectors along the channel dimension, the matching of multi-source data dimensions is improved. By calculating the cross-modal correlation degree, the quantitative accuracy of the intrinsic correlation between building changes, construction noise and work order risks is improved, and the pertinence of feature interaction is enhanced. By generating a unified fusion feature vector through weighted fusion processing, a collaborative discrimination basis is provided for the risk decision model, improving the accuracy of business object status assessment and the precision of early warning.

[0077] In some embodiments, performing visual structuring processing on the visual image to be processed to obtain a target region mask image includes: performing illumination equalization processing on the visual image to be processed to obtain an illumination compensation image; performing background region segmentation processing on the illumination compensation image to obtain an initial mask image; and performing edge fidelity filtering processing on the initial mask image to obtain a target region mask image.

[0078] Specifically, illumination equalization processing can be a process of adjusting the overall brightness distribution of an image to eliminate the effects of uneven illumination. This can improve the visibility of backlit, nighttime, or shadowed areas and enhance the reliability of subsequent processing steps. Illumination equalization processing can be achieved through Retinex-type algorithms to realize global illumination compensation. The illumination-compensated image can be the output image obtained through illumination equalization processing, which can provide visual data with uniform illumination conditions, laying the foundation for background region segmentation processing.

[0079] Furthermore, illumination equalization processing can be achieved through a multi-scale retinal theoretical model. By estimating the illumination components and normalizing them, local areas that are too dark or too bright can be eliminated, allowing the outline and structural features of the visual image to be processed to maintain clarity under complex lighting conditions.

[0080] In addition, background region segmentation can be a pixel-level classification process that separates the foreground of the business object from the irrelevant background in the illumination-compensated image. This can remove interfering elements such as trees, sky, and vehicles, and focus on the business object area. The initial mask image can be a binary segmentation result, in which the target area can be marked with a highlight pixel. This can initially mark the spatial location of the business object state and provide input for edge optimization.

[0081] Furthermore, background region segmentation can be achieved using the semantic segmentation model of the U-Net++ architecture. Multi-scale features can be extracted through the encoder-decoder structure, and deep and shallow features can be fused through skip connections to achieve segmentation of the edges and shapes of business objects. The encoder part can extract features through convolutional layer downsampling, and the decoder part can restore spatial resolution through deconvolutional layer upsampling. Then, the probability map of each pixel belonging to the business object region can be output through the activation function (Softmax).

[0082] In addition, edge-fidelity filtering is a filtering process that can smooth image noise while preserving the integrity of the target boundary. This can eliminate island noise and burrs in the initial mask image, while maintaining the sharpness of the object's outline.

[0083] Furthermore, edge-fidelity filtering can be achieved through gradient-domain-based bilateral filter processing. By combining spatial and domain kernel functions, the gradient response of edge regions can be enhanced while smoothing uniform regions. The spatial kernel function can be weighted according to pixel distance, and the domain kernel function can be weighted according to pixel intensity differences, ensuring that the boundaries of business objects are not blurred during the filtering process.

[0084] For example, in the scenario of identifying illegal buildings in residential communities, the visual image to be processed can be subjected to illumination equalization. By decomposing the reflection component and illumination component through a multi-scale retinal model, gamma correction and histogram matching are performed on the illumination component to generate an illumination-compensated image that makes the details of the roof structure and the covered area visible. The illumination-compensated image can then be processed for background region segmentation. A pre-trained U-Net++ model can be used to separate the roof building area from the sky and tree background, outputting an initial mask image. The roof outline can be marked with white pixels while the background can be black. The initial mask image can then be processed for edge fidelity filtering. A bilateral filtering algorithm is used to iteratively process the noise points in the mask while preserving the right angle and straight line features of the roof edge, resulting in a target region mask image for subsequent ViT model difference comparison.

[0085] According to the technical solution provided in the embodiments of this disclosure, by sequentially performing illumination equalization, background segmentation and edge fidelity filtering on the visual image, the interference of ambient light is eliminated, the image quality is improved, the accuracy of separating the building target from the background is improved, the integrity of the mask edge is optimized to ensure the accuracy of feature extraction, the robustness and accuracy of business object status assessment are improved, and the visual misjudgment rate is reduced in complex scenarios such as backlighting and occlusion.

[0086] In some embodiments, the environmental audio data is subjected to feature extraction processing to obtain an audio feature vector, including: performing steady-state noise suppression processing on the environmental audio data to obtain noise-reduced frequency data; performing joint feature extraction processing on the noise-reduced frequency data to obtain the spectral features corresponding to the environmental audio data; and performing principal component analysis processing on the spectral features to obtain the audio feature vector.

[0087] Specifically, steady-state noise suppression processing can be a wavelet transform-based filtering process, which can be a signal processing process that removes continuous background noise through time-frequency analysis, thereby reducing environmental interference and improving audio clarity; the noise-reduced audio data can be an audio signal obtained through noise suppression processing, and the noise-reduced audio data can be in the form of data that retains the main acoustic events while reducing background noise.

[0088] Furthermore, for example, in the scenario of monitoring illegal construction in a residential community, the audio of the surrounding environment of the building can be collected in real time by a microphone, and steady-state noise such as fan operation and traffic flow can be identified and filtered by wavelet transform algorithm, and the noise-reduced audio stream can be output, i.e. noise-reduced audio data.

[0089] In addition, joint feature extraction processing can be a parallel computation of multiple acoustic features, which can simultaneously extract time-domain and frequency-domain features to comprehensively describe audio properties. This can capture multidimensional representations of illegal noise. Spectral features can be characteristics used to characterize the frequency distribution of audio. The form of spectral features can be frequency domain parameters generated by short-time Fourier transform and Mel frequency cepstral coefficients, which can quantify the spectral energy and patterns of sound.

[0090] Furthermore, for example, during the identification of illegal construction, features can be extracted from the noise reduction frequency data, and the MFCC coefficients and STFT spectrum can be calculated simultaneously to generate a 128-dimensional spectral feature vector to characterize the frequency domain characteristics of the noise.

[0091] In addition, principal component analysis can be a statistical dimensionality reduction process, specifically by mapping high-dimensional features to a low-dimensional space through orthogonal transformation, thereby reducing feature redundancy and improving computational efficiency.

[0092] Furthermore, for example, in property management system applications, spectral features can be processed through principal component analysis to compress 128-dimensional features into a 64-dimensional audio feature vector, which can then be input into a classifier combining CNN and LSTM for illegal noise identification, achieving efficient detection of construction events.

[0093] According to the technical solution provided in this disclosure, environmental interference is eliminated through steady-state noise suppression processing, which improves the quality of audio data and ensures the accuracy of feature extraction; multi-dimensional acoustic features are captured through joint feature extraction processing, which enhances the characterization ability of illegal noise; feature dimensions are optimized through principal component analysis processing, which improves the efficiency of model training and inference; the robustness of business object status assessment is enhanced, and the monitoring reliability in complex audio environments is improved.

[0094] In some embodiments, feature extraction processing is performed on the business data to be processed to obtain structured keyword data, including: normalizing the fields of the business data to be processed to obtain standardized business data; calculating keyword weights on the standardized business data to obtain a keyword weight sequence; and filtering the keyword weight sequence based on a preset weight threshold to obtain structured keyword data.

[0095] Specifically, field normalization can be achieved by mapping attribute fields with different dimensions, formats, and naming conventions in the business data to a unified dimension, unified namespace, and unified data type. This can be accomplished through a combination of minimum-maximum normalization and standardization concatenation. Numerical fields can be linearly scaled to the [0,1] interval; translation-scaling transformations can be performed using a Gaussian distribution with a mean of 0 and a variance of 1 as the target domain; categorical fields can be one-hot encoded and then normalized using the L2 norm of sparse vectors; and time-related fields can be linearly compressed to the [0,1] interval using relative timestamps. This results in a set of floating-point multidimensional vectors with zero mean, unit variance, orthogonal dimensions, and no dimension conflicts—the standardized business data. This standardized business data can be used to eliminate dimensional differences and distribution skewness between heterogeneous fields, ensuring that the information entropy contribution of each feature dimension to the TF-IDF value is on a comparable scale during the keyword weight calculation stage.

[0096] In addition, keyword weight calculation processing can refer to the process of calculating the importance weight of keywords in standardized business data using the TF-IDF algorithm. Specifically, it can involve statistically analyzing the frequency and inverse document frequency of keywords in a document set, thereby quantifying the representativeness of keywords and identifying key terms related to the status of business objects. Keyword weight sequence can refer to an ordered list containing keywords and their corresponding weights. The form of this keyword weight sequence can be a serialized representation of weight values, which can be used to store the importance ranking of keywords.

[0097] Furthermore, the keyword weight calculation process can also include word segmentation and stop word removal of the work order text to extract effective keywords such as "construction waste" or "wall demolition," etc., without limitation here, and weight values ​​can be calculated based on word frequency distribution.

[0098] In addition, the preset weight threshold can be a pre-defined numerical limit. This preset weight threshold can be used to distinguish between important and unimportant keywords. It can be a configurable filtering standard, used to control the retention range of keywords, improve data quality, and the filtering process can remove keywords with weights lower than the preset weight threshold from the keyword weight sequence according to the preset weight threshold. It can also be used to retain high-weight keywords and reduce redundant information.

[0099] For example, in the identification of illegal construction in residential communities, the business data to be processed may include work order records obtained from the property system, such as garbage collection work orders and maintenance requests. The work order dates and types can be standardized into a standard format through field normalization to generate standardized business data. Then, keyword weight calculation can be performed to extract keywords such as "illegal construction" and calculate their TF-IDF weights to obtain a keyword weight sequence. Low-weight keywords can be filtered based on preset weight thresholds to obtain structured keyword data.

[0100] According to the technical solution provided in this disclosure, by performing field normalization on the business data to be processed, data format inconsistencies are eliminated, and the reliability of subsequent processing is improved; by performing keyword weight calculation, the importance of keywords is quantified, and the targeting of feature extraction is enhanced; by performing filtering based on preset weight thresholds, noise data interference is reduced, the quality of structured keyword data is optimized, the accuracy and efficiency of business object status assessment are improved, and the precise implementation of multimodal data fusion decision-making is supported.

[0101] In some embodiments, the channel feature set is weighted and fused using an association weight matrix to obtain a fused feature vector, including: weighting the channel feature set based on the association weight matrix to obtain a weighted feature subset; performing information filtering on the weighted feature subset to obtain a compressed feature set; and performing feature concatenation on the compressed feature set to obtain a fused feature vector.

[0102] Specifically, importance weighting can be a process of element-wise weighting or matrix multiplication of the correlation weight matrix and the channel feature set, which can highlight high-weight features and suppress low-weight features, thereby improving the discriminative power of the fused features. The weighted feature subset can be the output result of importance weighting, specifically a set of feature vectors adjusted by weights, which can be used to retain the weighted feature information for subsequent processing.

[0103] In addition, information filtering can be a process of removing redundant or low-importance features through preset threshold filtering or dimensionality reduction, thereby reducing feature dimensionality and eliminating noise, improving the efficiency and robustness of subsequent processing. Compressed feature sets can be feature sets obtained by reducing dimensionality, which can be used to retain core information and simplify data structures.

[0104] In addition, feature concatenation can be a process of concatenating multiple feature vectors along a specified dimension to form a new vector, thereby integrating multi-source feature information to form a unified high-dimensional representation.

[0105] For example, in the scenario of identifying illegal buildings in residential communities, the channel feature set can be weighted and fused using an association weight matrix. This allows for the extraction of the channel feature set from multimodal data, including visual features of buildings, audio features of illegal noise, and business features related to work orders and risks. The association weight matrix, generated based on a cross-modal attention mechanism, can be used to weight the channel feature set according to importance, with visual features having higher weights, audio features next, and business features having lower weights, resulting in a weighted feature subset. This weighted feature subset can then undergo information filtering, removing features with weights below a certain threshold to obtain a compressed feature set. Finally, the compressed feature set can be concatenated, connecting all remaining features along the vector dimension to form a fused feature vector.

[0106] According to the technical solution provided in this disclosure, by weighting the importance of the association weight matrix, the weights of multimodal features are dynamically adjusted, which enhances the expression of key features and improves the discriminative ability of fused features; by information filtering, redundant and noisy features are eliminated, improving computational efficiency and reducing the risk of overfitting; by feature splicing, multi-source feature information is integrated to form a unified feature representation, which enhances robustness and accuracy, and improves the accuracy and timeliness of early warning for business object status assessment.

[0107] All of the above-mentioned optional technical solutions can be combined in any way to form optional embodiments of this disclosure, and will not be described in detail here.

[0108] Figure 3 This is a schematic diagram of another business object status evaluation method provided in this disclosure embodiment. For example... Figure 3 As shown, the business object status assessment method includes: Multimodal data acquisition: Visual data (visual images to be processed), audio data (environmental audio data), and business data (business data to be processed) are acquired through cameras, drones, microphones, and property system interfaces. Data preprocessing: Cleaning, enhancing, and extracting features from multi-source data to output standardized features; Multimodal AI recognition: Separately identifies construction violations, material transportation, illegal noise, and work order risks; Multimodal fusion decision-making: Integrates multi-dimensional identification results to output violation risk levels and early warning information; Closed-loop iterative update: Based on the verification and feedback from management personnel, the model parameters and benchmark feature library are dynamically updated.

[0109] According to the technical solution provided in this disclosure, a target region mask image is obtained by performing visual structuring processing on the visual image to be processed; feature extraction processing is performed on the environmental audio data and the business data to be processed to obtain audio feature vectors and structured keyword data; multimodal recognition processing is performed on the target region mask image, audio feature vectors, and structured keyword data respectively to obtain visual feature vectors, audio feature vectors, and text feature vectors; cross-modal attention fusion processing is performed on the visual feature vectors, audio feature vectors, and text feature vectors to obtain fused feature vectors; gradient boosting processing is performed on the fused feature vectors to obtain the target business object state evaluation result, thereby improving the accuracy of business object state evaluation, enhancing the understanding of complex scenarios, improving the efficiency of multimodal data fusion and the robustness of evaluation results, improving adaptability, and enhancing dynamic iteration capability.

[0110] Figure 4 This is a schematic diagram of a multimodal data acquisition and preprocessing method provided in an embodiment of this disclosure. Figure 4 As shown, the multimodal data acquisition and preprocessing method includes: The data collection process, in conjunction with actual application scenarios, can specifically include: 1. Visual image data: Fixed cameras (supporting infrared night vision / rain mode) can be deployed at key locations such as entrances and exits, building perimeters, and rooftops to collect images of building exteriors and videos of personnel and material transportation; drones take weekly aerial photos to obtain high-altitude images of buildings; mobile inspection terminals collect images / videos of concealed areas; 2. Audio data: Environmental audio can be collected by deploying microphones at building unit entrances and public areas; 3. Property business data: It can connect to the property system's work order data (garbage collection, maintenance, complaints) and visitor access data (identity information, facial image, access time, vehicle information) through the interface.

[0111] The preprocessing stage specifically includes: 1. Visual data preprocessing: Image enhancement: Retinex algorithm optimizes backlight / nighttime images, Gaussian filtering removes noise; Semantic segmentation: U-Net++ model segments building areas, removing background elements such as trees and vehicles; Video frame extraction: Keyframes are extracted from entrance / exit videos according to preset frequencies; 2. Audio data preprocessing: Wavelet transform to filter background noise; extract MFCC and STFT features to generate feature vectors; 3. Business data preprocessing: Work order data: Standardized fields, TF-IDF keyword extraction (such as "construction waste" and "wall demolition"); Visitor data: Face image alignment and normalization, rule engine to filter abnormal data (such as multiple passages in a single day, multiple people traveling together without registering materials).

[0112] According to the technical solution provided in this disclosure, a full-domain perception network is constructed using fixed cameras, drones, mobile inspection terminals, microphones, and property system interfaces, enabling the simultaneous acquisition of three types of heterogeneous data: visual, audio, and business data. Retinex illumination equalization and Gaussian filtering are used for global illumination compensation and spatial noise reduction of backlight, night vision, and rain / fog images. A U-Net++ nested dense skip connection mechanism is combined to achieve pixel-level building region segmentation, improving the accuracy of removing dynamic background interference such as trees and vehicles. A keyframe extraction strategy reduces video redundancy, ensuring that visual features are both clear and representative. In the audio link, wavelet thresholding is used to remove… The system suppresses non-steady-state environmental noise and extracts MFCC Mel-frequency cepstral coefficients and STFT short-time Fourier transform coefficients in parallel, improving robustness. Unstructured chemical text is transformed into highly discriminative keyword sequences such as "construction waste" and "wall demolition" through field normalization and TF-IDF weight calculation. Affine alignment and normalization encoding are performed on facial images, and a rule engine is deployed to filter out abnormal events such as multiple passages in a single day and unregistered group travel, ensuring the credibility of business features. This improves the quality, consistency, and interpretability of multimodal data, enhances the reliability of cross-modal information fusion, and improves the accuracy and real-time performance of business object status assessment.

[0113] Figure 5 This is a schematic diagram of a multimodal artificial intelligence recognition method provided in an embodiment of this disclosure. Figure 5 As shown, this multimodal artificial intelligence recognition method includes: The specific processing flow of the building violation visual recognition submodule is as follows: The following is an example of a dual-model approach combining "ViT differential comparison with YOLOv9 precise detection" and specific application scenarios: 1. Construction of the benchmark feature library: Input the original design drawings and compliant images of the community, and the ViT model (which can be set to 12 layers of encoder and 12 attention heads) learns the standard structural features of the buildings (floor height, balcony shape, roof outline) to establish a benchmark library for each building; 2. Real-time difference comparison: The ViT model extracts the current structural features of real-time images (fixed camera / drone) and calculates the structural difference (contour deviation, area increase ratio) with the benchmark library. 3. Accurate detection in complex scenes: The YOLOv9 model with enhanced attention mechanism (confidence threshold can be 0.85) performs secondary detection on suspected illegal areas (difference greater than or equal to 10%), identifies the edges and material features of the covered structure, and eliminates environmental interference; 4. Result judgment: If the difference is greater than or equal to 10% and the YOLOv9 confidence level is greater than or equal to 0.85, output a preliminary warning of building violation.

[0114] The personnel and material identification submodule, combined with an application scenario example, has the following specific processing flow: 1. Personnel Role Recognition: It can recognize visitors' faces and clothing features (safety helmet, work clothes), and determine their roles (construction worker / building material transport personnel / ordinary visitor) based on registration information; 2. Material Transportation Inspection: The YOLOv9 model detects the type of transport vehicle (truck / van), the characteristics of building materials (cement bags, steel), and counts the quantity of materials; 3. Anomaly Matching Analysis: Compare personnel roles, number of people traveling together, and quantity of materials (e.g., 5 people traveling together with a large quantity of unreported steel is considered anomaly), and output an alert for illegal material transportation.

[0115] The illegal noise identification submodule, combined with an application scenario example, has the following specific processing flow: 1. Noise classifier training: CNN combined with LSTM model inputs compliant / illegal noise samples (up to 1300 samples) to learn the features of illegal noise (frequency of 500-2000 Hz, energy peak of greater than or equal to 85 dB). 2. Real-time monitoring: Audio feature vectors are input into a classifier to identify illegal noise and record its time and duration; 3. Linked verification: If noise persists for more than 30 minutes outside of the reporting period and there are records of renovation workers entering the corresponding area, an early warning of illegal construction will be issued.

[0116] The work order risk identification submodule, combined with an application scenario example, has the following specific processing flow: 1. Work order feature extraction: Semantic features of work orders can be extracted using the BERT model, and feature vectors can be constructed by combining collection frequency and region; 2. Risk Assessment: The LightGBM model (which can contain 100 decision trees and has a learning rate of 0.05) can be used to output the risk level (low / medium / high). Construction waste removal in the same area more than twice within 3 days is considered high risk. 3. Correlation analysis: High-risk work order areas are matched with visual recognition and noise monitoring areas to improve the priority of early warning.

[0117] According to the technical solution provided in this disclosure, structural changes are locked by visual differential detection to accurately detect and locate illegal additions; personnel roles and transported materials are identified simultaneously and their legality is automatically compared; abnormal frequency noise is monitored and a construction alarm is triggered if it continues; work order semantics are analyzed to assess risks and multimodal information is linked to improve the priority of early warnings; finally, four types of violation results—building, transportation, construction, and work order—are integrated and output, thereby improving the processing speed and recognition accuracy of business object status assessment results, enhancing the judgment capability in multi-role, multi-material, and multi-noise scenarios, and improving the level of automation, intelligence, and refined management.

[0118] Figure 6 This is a schematic diagram of a multimodal fusion decision-making and closed-loop iterative method provided in an embodiment of this disclosure. Figure 6 As shown, the multimodal fusion decision-making and closed-loop iterative method includes: Using a cross-modal attention fusion network combined with an improved XGBoost model and an application scenario as an example, the specific processing flow is as follows: Feature fusion: Visual features (building differences, material confidence), audio features (noise confidence, duration), and business features (work order risk level, personnel matching degree) can be fused into a 1024-dimensional vector; Risk Decision: The XGBoost model (150 iterations, learning rate 0.03) takes the fused vector as input and outputs the risk level (high level can be greater than or equal to 0.8, medium level can be greater than or equal to 0.6 and less than 0.8, and low level can be less than 0.6) and the judgment criteria. Warning output: Medium and high risk information is pushed to the management personnel terminal, including the area of ​​violation, image / audio evidence, and related data.

[0119] The closed-loop iteration mechanism, combined with an application scenario example, has the following specific processing flow: Early warning feedback: Management personnel verify the early warning results (whether true or false) and provide feedback; Model update: The parameters of each recognition sub-module can be updated based on feedback through online learning algorithms (such as optimizing feature weights for misjudged cases). Baseline library update: Images of compliant modifications (such as balcony enclosure after reporting) are updated to the baseline library to avoid subsequent misjudgments; Timing logic: Early warning push (time scale is 0, which can be represented as T0) → verification feedback (T0+2 hours) → model update (T0+24 hours) → benchmark library update (T0+30 days).

[0120] According to the technical solution provided in this disclosure, a cross-modal attention network is used to compress visual difference confidence audio violation confidence and work order personnel risk semantics into a unified semantic vector. An improved XGBoost is used to output high, medium and low risk levels and obtain interpretable judgments. The warning immediately pushes image and audio evidence packages to the management terminal. By synchronously updating the visual difference benchmark library and audio classification boundary, a closed loop of discovery, verification, correction and relearning is formed, which improves the accuracy of warnings and suppresses false alarms, and enhances the adaptive capability and long-term optimization mechanism.

[0121] The following are embodiments of the apparatus disclosed herein, which can be used to execute embodiments of the method disclosed herein. For details not disclosed in the apparatus embodiments of this disclosure, please refer to the embodiments of the method disclosed herein.

[0122] Figure 7 This is a schematic diagram of a business object status assessment device provided in an embodiment of this disclosure. Figure 7 As shown, the business object status assessment device includes: The first processing module 701 is used to perform visual structuring processing on the visual image to be processed to obtain a target region mask image. The second processing module 702 is used to perform feature extraction processing on the environmental audio data and the business data to be processed to obtain audio feature vectors and structured keyword data. The third processing module 703 is used to perform multimodal recognition processing on the target area mask image, audio feature vector and structured keyword data respectively to obtain visual feature vector, audio feature vector and text feature vector; The fourth processing module 704 is used to perform cross-modal attention fusion processing on the visual feature vector, audio feature vector and text feature vector to obtain a fused feature vector; The fifth processing module 705 is used to perform gradient boosting on the fused feature vector to obtain the target business object status evaluation result.

[0123] According to the technical solution provided in this disclosure, a target region mask image is obtained by performing visual structuring processing on the visual image to be processed; feature extraction processing is performed on the environmental audio data and the business data to be processed to obtain audio feature vectors and structured keyword data; multimodal recognition processing is performed on the target region mask image, audio feature vectors, and structured keyword data respectively to obtain visual feature vectors, audio feature vectors, and text feature vectors; cross-modal attention fusion processing is performed on the visual feature vectors, audio feature vectors, and text feature vectors to obtain fused feature vectors; gradient boosting processing is performed on the fused feature vectors to obtain the target business object state evaluation result, thereby improving the accuracy of business object state evaluation, enhancing the understanding of complex scenarios, improving the efficiency of multimodal data fusion and the robustness of evaluation results, improving adaptability, and enhancing dynamic iteration capability.

[0124] In some embodiments, the fifth processing module 705 is specifically used to: perform confidence calculation processing on the fused feature vector to obtain a confidence label; perform weight adaptive processing on the fused feature vector based on the confidence label to obtain a fused feature correction vector; and perform iterative evaluation processing on the fused feature correction vector to obtain the target business object status evaluation result.

[0125] In some embodiments, the fourth processing module 704 is specifically used to perform channel dimension alignment processing on the visual feature vector, audio feature vector and text feature vector to obtain a channel feature set; perform cross-modal correlation degree calculation processing on the channel feature set to obtain a correlation weight matrix; and perform weighted fusion processing on the channel feature set through the correlation weight matrix to obtain a fused feature vector.

[0126] In some embodiments, the first processing module 701 is specifically used to perform illumination equalization processing on the visual image to be processed to obtain an illumination compensated image; perform background region segmentation processing on the illumination compensated image to obtain an initial mask image; and perform edge fidelity filtering processing on the initial mask image to obtain a target region mask image.

[0127] In some embodiments, the second processing module 702 is specifically used to perform steady-state noise suppression processing on the environmental audio data to obtain noise-reduced frequency data; perform joint feature extraction processing on the noise-reduced frequency data to obtain the spectral features corresponding to the environmental audio data; and perform principal component analysis processing on the spectral features to obtain the audio feature vector.

[0128] In some embodiments, the second processing module 702 is specifically used to perform field normalization processing on the business data to be processed to obtain standardized business data; perform keyword weight calculation processing on the standardized business data to obtain a keyword weight sequence; and perform filtering processing on the keyword weight sequence based on a preset weight threshold to obtain structured keyword data.

[0129] In some embodiments, the channel feature set is weighted and fused using an association weight matrix to obtain a fused feature vector. Specifically, the channel feature set is weighted based on the association weight matrix to obtain a weighted feature subset; the weighted feature subset is filtered to obtain a compressed feature set; and the compressed feature set is concatenated to obtain a fused feature vector.

[0130] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this disclosure.

[0131] Figure 8 This is a schematic diagram of the electronic device 8 provided in an embodiment of this disclosure. Figure 8 As shown, the electronic device 8 of this embodiment includes a processor 801, a memory 802, and a computer program 803 stored in the memory 802 and executable on the processor 801. When the processor 801 executes the computer program 803, it implements the steps in the various method embodiments described above. Alternatively, when the processor 801 executes the computer program 803, it implements the functions of each module / unit in the various device embodiments described above.

[0132] Electronic device 8 can be a desktop computer, laptop, handheld computer, cloud server, or other electronic device. Electronic device 8 may include, but is not limited to, processor 801 and memory 802. Those skilled in the art will understand that... Figure 8 This is merely an example of electronic device 8 and does not constitute a limitation on electronic device 8. It may include more or fewer components than shown, or different components.

[0133] The processor 801 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0134] The memory 802 can be an internal storage unit of the electronic device 8, such as a hard disk or RAM of the electronic device 8. The memory 802 can also be an external storage device of the electronic device 8, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc., equipped on the electronic device 8. The memory 802 can also include both internal and external storage units of the electronic device 8. The memory 802 is used to store computer programs and other programs and data required by the electronic device.

[0135] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0136] If integrated modules / units are implemented as software functional units and sold or used as independent products, they can be stored in a readable storage medium (e.g., a computer-readable storage medium). Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program may include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable storage medium may include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0137] The above embodiments are only used to illustrate the technical solutions of this disclosure, and are not intended to limit it. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure, and should all be included within the protection scope of this disclosure.

Claims

1. A method of assessing the state of a business object, characterized by, include: Visual structuring is performed on the visual image to be processed to obtain a mask image of the target region; The environmental audio data and the business data to be processed are subjected to feature extraction processing to obtain audio feature vectors and structured keyword data; Multimodal recognition processing is performed on the target region mask image, the audio feature vector, and the structured keyword data to obtain visual feature vectors, audio feature vectors, and text feature vectors; The visual feature vector, audio feature vector, and text feature vector are subjected to cross-modal attention fusion processing to obtain a fused feature vector; Gradient boosting is performed on the fused feature vector to obtain the target business object state evaluation result.

2. The business object state evaluation method of claim 1, wherein, The gradient boosting process performed on the fused feature vector to obtain the target business object state evaluation result includes: The confidence score is calculated on the fused feature vector to obtain the confidence score label; Based on the confidence labels, the fused feature vector is subjected to weight adaptive processing to obtain the fused feature correction vector; The fusion feature correction vector is iteratively evaluated to obtain the state evaluation result of the target business object.

3. The business object state evaluation method of claim 1, wherein, The process of performing cross-modal attention fusion processing on the visual feature vector, audio feature vector, and text feature vector to obtain a fused feature vector includes: The visual feature vector, the audio feature vector, and the text feature vector are aligned by channel dimension to obtain a channel feature set. The cross-modal correlation degree of the channel feature set is calculated to obtain the correlation weight matrix; The channel feature set is weighted and fused using the correlation weight matrix to obtain the fused feature vector.

4. The business object state evaluation method of claim 1, wherein, The step of performing visual structuring processing on the visual image to be processed to obtain a target region mask image includes: The visual image to be processed is subjected to illumination equalization processing to obtain an illumination-compensated image; The illumination-compensated image is subjected to background region segmentation processing to obtain an initial mask image; The initial mask image is subjected to edge-fidelity filtering to obtain the target region mask image.

5. The business object state evaluation method of claim 1, wherein, The step of extracting features from environmental audio data to obtain audio feature vectors includes: The environmental audio data is subjected to steady-state noise suppression processing to obtain noise-reduced audio data; The noise-reduced audio data is subjected to joint feature extraction processing to obtain the spectral features corresponding to the environmental audio data; Principal component analysis is performed on the spectral features to obtain the audio feature vector.

6. The business object state evaluation method of claim 1, wherein, The step of performing feature extraction processing on the business data to be processed to obtain structured keyword data includes: The business data to be processed is subjected to field normalization to obtain standardized business data; The standardized business data is processed by keyword weight calculation to obtain a keyword weight sequence; The keyword weight sequence is filtered based on a preset weight threshold to obtain the structured keyword data.

7. The business object status assessment method according to claim 3, characterized in that, The step of weighting and fusing the channel feature set using the correlation weight matrix to obtain the fused feature vector includes: The channel feature set is weighted based on the correlation weight matrix to obtain a weighted feature subset; The weighted feature subset is subjected to information filtering to obtain a compressed feature set; The compressed feature set is subjected to feature concatenation processing to obtain the fused feature vector.

8. A business object status assessment device, characterized in that, include: The first processing module is used to perform visual structuring on the visual image to be processed to obtain a mask image of the target region. The second processing module is used to perform feature extraction processing on environmental audio data and business data to be processed, to obtain audio feature vectors and structured keyword data. The third processing module is used to perform multimodal recognition processing on the target region mask image, the audio feature vector and the structured keyword data respectively to obtain visual feature vector, audio feature vector and text feature vector; The fourth processing module is used to perform cross-modal attention fusion processing on the visual feature vector, audio feature vector and text feature vector to obtain a fused feature vector; The fifth processing module is used to perform gradient boosting processing on the fused feature vector to obtain the target business object state evaluation result.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.

10. A readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images