Welfare institution service evaluation and system based on ai vision multi-modal large model

Abstract

The application provides a welfare institution service evaluation method and system based on an AI vision multi-modal large model, real-time collection of monitoring video streams of a welfare institution, desensitization processing of private bedroom area videos, extraction of image visual features, text prompt features, and reliance on knowledge features to jointly identify the nursing behavior types of the welfare institution, break through the limitations of traditional skeleton threshold algorithms, realize full-coverage identification of implicit behaviors, rely on multi-modal semantic understanding + standard knowledge constraints, get rid of the dependence on limb features, accurately identify non-standard behaviors, and completely solve the industry pain points of large-area missed detection of traditional solutions.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, multimodal large models and intelligent supervision of elderly care, and more specifically, to a service evaluation method and system for welfare institutions based on AI vision multimodal large models. Background Technology

[0002] Currently, commercial intelligent monitoring products for welfare institutions in China generally adopt a judgment architecture that combines YOLO-Pose skeletal key point detection with manually fixed thresholds. This architecture relies solely on superficial physical features such as human skeletal coordinates, limb movement speed, and action duration to determine behavior, which presents inherent technical bottlenecks and cannot meet the refined, compliant, and intelligent monitoring needs of welfare institutions. The main pain points are as follows:

[0003] First, the recognition logic is rigid, resulting in widespread missed detections of service behaviors. Traditional solutions rely solely on limb movement features to determine behavior, lacking the ability to understand contextual semantics, and are completely ineffective in recognizing behaviors without obvious limb movements.

[0004] Second, the generalization ability is weak, and it cannot identify new types of irregular behaviors. Traditional models can only identify pre-labeled actions, and have no zero-sample identification capability for newly emerging covert behaviors in elderly care supervision.

[0005] In addition, existing general-purpose open-source VLM multimodal large models are not adapted to the specific scenarios of elderly care supervision, are not connected to elderly care standards, lack compliance constraints for model inference, have highly subjective and unreliable recognition results, and cannot be integrated with the evaluation system. Summary of the Invention

[0006] To address the technical shortcomings of traditional skeletal thresholding algorithms that rely solely on limb features and cannot recognize behaviors without limb contact, this paper presents a service evaluation method and system for welfare institutions based on an AI-powered visual multimodal large model, enabling semantic-level scene understanding and legal normative judgment.

[0007] According to a first aspect of the present invention, a method for evaluating welfare institution services based on an AI visual multimodal large model is provided, comprising:

[0008] Step 1: Real-time acquisition of surveillance video streams from welfare institutions; desensitization processing of video from private living areas to obtain desensitized images;

[0009] Step 2: Input the desensitized image into the trained VLM multimodal large model, generate image visual features through the visual encoder, and encode the preset task prompt words into text prompt features through the text embedding layer;

[0010] Step 3: Construct an offline structured regulatory norms knowledge base, which contains vectorized normative text fragments and their semantic tags;

[0011] Step 4: Perform global mean pooling on the image visual features to generate an image semantic retrieval vector, and extract scene semantic keywords corresponding to the image visual features; based on the image semantic retrieval vector and the scene semantic keywords, adopt a two-layer retrieval mechanism of keyword hard matching and vector semantic similarity soft recall to recall matching standard entries from the knowledge base, and calculate the matching confidence score;

[0012] Step 5: Concatenate and fuse the image visual features, the text prompt features, and the feature vectors of the recalled standard entries to obtain enhanced fused features;

[0013] Step 6: Input the enhanced fusion features into the large language decoder in the VLM multimodal large model for inference to obtain the nursing behavior type;

[0014] Step 7: Based on the matching confidence score, nursing behaviors are classified into different levels, and the welfare institution is evaluated based on the identified nursing behavior type and the duration of continuous behavior.

[0015] Optionally, step 1 involves real-time acquisition of surveillance video streams from the welfare institution, and desensitizing the video footage from private living areas to obtain desensitized images, including:

[0016] The system collects real-time surveillance video streams from welfare institutions, determines the scene area of ​​the video stream, and if it is in a public area, no processing is performed on the video stream; if it is in a private room, human targets are located through target detection, the region of interest (ROI) is adaptively cropped, and pixel mean desensitization processing is performed on the face area within the ROI, and the desensitized image is stitched together.

[0017] Optionally, in step 1, if the location is a private living area, the human target is located through target detection, the region of interest (ROI) is adaptively cropped, and pixel mean desensitization processing is performed on the face region within the ROI to obtain a desensitized image, including:

[0018] For videos of private living areas, human targets within the private living area are detected using a target detection model to obtain human bounding boxes;

[0019] The human bounding box is expanded according to a preset adaptive magnification factor to generate an initial ROI cropping box. The boundary constraints of the initial ROI cropping box are then corrected by combining the width and height of the video frame image to obtain an effective interactive cropping box.

[0020] Based on the effective interactive cropping box, a ROI sub-image is cropped from the video frame image. The face region within the ROI sub-image is detected, the average pixel value of each channel in the face region is calculated, and the average pixel value is assigned to all pixels in the face region. The pixels in the face region are desensitized, and the desensitized face region and non-privacy background region are stitched together to obtain a complete desensitized image.

[0021] Optionally, the VLM multimodal large model in step 2 adopts a lightweight deployment with a frozen pre-trained backbone and low-rank adaptation fine-tuning, including:

[0022] Freeze the pre-trained core weight matrix W0 of the VLM multimodal large model, introduce low-rank up projection matrix A and low-rank down projection matrix B, and fine-tune the inference weights of the VLM multimodal large model as follows: , where the rank r of B and A is much smaller than the dimension of W0;

[0023] During the training of the VLM multimodal large model, the gradient is backpropagated based on the loss function of the nursing behavior recognition task. Only the up-projection matrix A and the low-rank down-projection matrix B are updated. The pre-trained core weight matrix W0 is frozen throughout the process. The iteration continues until the loss converges, and the trained VLM multimodal large model is obtained.

[0024] Optionally, step 3 involves constructing an offline structured regulatory norm knowledge base, which includes vectorized normative text fragments and their semantic tags, including:

[0025] Collect the pre-set service safety standard texts and assessment rule texts for welfare institutions, classify each standard text according to the pre-set classification set, and obtain the classification label for each standard text;

[0026] Each standard text is sliced ​​according to a set text length threshold, and valid text segments containing descriptions of violations and mandatory constraints are selected to generate a set of standard slices.

[0027] Each specification slice is bound to at least one preset semantic tag and evaluation parameter;

[0028] For each standardized slice, a dense feature vector is generated, and a structured offline knowledge base containing classification labels, standardized slices, semantic labels, evaluation parameters, and dense feature vectors is constructed.

[0029] Optionally, in step 4, based on the image semantic retrieval vector and the scene semantic keywords, a two-layer retrieval mechanism of hard keyword matching and soft recall based on vector semantic similarity is used to recall matching standard entries from the knowledge base, including:

[0030] Extract scene semantic keywords corresponding to the visual features of the image, traverse the semantic tags in the knowledge base based on the scene semantic keywords, filter the matching standard slices, and obtain a hard matching set;

[0031] Calculate the similarity between the image semantic retrieval vector and the dense feature vector of each canonical slice in the knowledge base, and filter the K canonical slices with the highest similarity to obtain the soft recall set;

[0032] The hard matching set and the soft recall set are deduplicated to obtain the final valid specification slice.

[0033] Optionally, in step 4, calculating the matching confidence score includes:

[0034] Calculate the matching confidence score based on the maximum similarity in the soft recall set;

[0035] Step 5 involves concatenating and fusing the image visual features, the text prompt features, and the feature vectors of the recalled canonical entries to obtain enhanced fusion features, including:

[0036] Based on the final valid standardized slices, mean pooling is performed on the dense feature vectors of all valid standardized slices to generate knowledge features;

[0037] The image visual features, the text prompt features, and the knowledge features are concatenated and fused to generate enhanced fusion features.

[0038] Optionally, step 7, based on the matching confidence score, categorizes nursing behaviors into different levels, including:

[0039] When the matching confidence score is less than the low-risk threshold, it is determined to be normal behavior;

[0040] When the matching confidence score is greater than or equal to the high-risk threshold, it is determined to be a high-risk behavior;

[0041] When the matching confidence score is between the low-risk threshold and the high-risk threshold, it is judged as a low-risk suspected behavior and the log is retained for manual review.

[0042] According to a second aspect of the present invention, a welfare institution service evaluation system based on an AI visual multimodal large model is provided, comprising:

[0043] The desensitization processing module is used to collect real-time surveillance video streams from welfare institutions, desensitize videos from private living areas, and obtain desensitized images.

[0044] The feature extraction module is used to input the desensitized image into the trained VLM multimodal large model, generate image visual features through a visual encoder, and encode preset task prompt words into text prompt features through a text embedding layer;

[0045] A construction module is used to build an offline structured regulatory norms knowledge base, which contains vectorized normative text fragments and their semantic tags.

[0046] The matching module is used to perform global mean pooling on the visual features of the image to generate an image semantic retrieval vector, and to extract scene semantic keywords corresponding to the visual features of the image; based on the image semantic retrieval vector and the scene semantic keywords, a two-layer retrieval mechanism of hard keyword matching and soft recall based on vector semantic similarity is adopted to recall the matching standard entries from the knowledge base and calculate the matching confidence score.

[0047] The fusion module is used to concatenate and fuse the image visual features, the text prompt features, and the feature vectors of the recalled canonical entries to obtain enhanced fusion features;

[0048] The inference module is used to input the enhanced fusion features into the large language decoder in the VLM multimodal large model for inference to obtain the nursing behavior type;

[0049] The evaluation module is used to classify nursing behaviors into different levels based on the matching confidence score, and to evaluate welfare institutions based on the identified nursing behavior types and the duration of continuous behavior.

[0050] According to a third aspect of the present invention, an electronic device is provided, including a memory and a processor, wherein the processor is configured to implement the steps of a welfare institution service evaluation method based on an AI visual multimodal large model when executing a computer management program stored in the memory.

[0051] According to a fourth aspect of the present invention, a computer-readable storage medium is provided having a computer management program stored thereon, which, when executed by a processor, implements the steps of a welfare institution service evaluation method based on an AI visual multimodal large model.

[0052] This invention provides a service evaluation method and system for welfare institutions based on an AI-powered visual multimodal large model. By real-time acquisition of surveillance video streams from welfare institutions, and desensitizing videos from private living areas, the system extracts visual features, textual prompts, and knowledge features to jointly identify the types of nursing behaviors in welfare institutions. This method overcomes the limitations of traditional skeletal threshold algorithms, achieving full coverage identification of implicit behaviors. Relying on multimodal semantic understanding and standardized knowledge constraints, this invention eliminates reliance on limb features and can accurately identify non-standard behaviors, completely solving the industry pain point of large-scale missed detections in traditional solutions. Attached Figure Description

[0053] Figure 1 A flowchart of a service evaluation method for welfare institutions based on an AI visual multimodal large model, provided as an embodiment of the present invention;

[0054] Figure 2 The diagram below shows a structural block diagram of a welfare institution service evaluation system based on an AI visual multimodal large model, as provided in one embodiment of the present invention.

[0055] Figure 3 A schematic diagram of the hardware structure of a possible electronic device provided by the present invention;

[0056] Figure 4 This is a schematic diagram of the hardware structure of a possible computer-readable storage medium provided by the present invention. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. In addition, the technical features of the various embodiments or individual embodiments provided by the present invention can be arbitrarily combined with each other to form feasible technical solutions. Such combinations are not constrained by the order of steps and / or structural composition patterns, but must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0058] Figure 1 This diagram illustrates a flowchart of a service evaluation method for welfare institutions based on an AI-powered visual multimodal large model, according to an embodiment of the present invention. Figure 1 As shown, the method includes the following steps:

[0059] Step 1: Real-time acquisition of surveillance video streams from welfare institutions, and desensitization processing of videos from private living areas to obtain desensitized images.

[0060] Understandably, there is an inherent conflict between privacy protection and computational power consumption. Traditional regulatory solutions either use full-screen upload for inference, which carries the risk of leaking private bedroom footage and fails to meet privacy compliance requirements. In this embodiment of the invention, multiple surveillance video streams within the facility are collected in real time, and the collected video streams are preprocessed and privacy-partitioned for cropping. Specifically, the multiple surveillance video streams are processed by scene partitioning: public areas (corridors, canteens, activity rooms) are fed into the model in full screen, while private bedroom areas only have their interaction-related ROI images cropped, and irrelevant environmental and facial areas are masked, achieving privacy-level preprocessing; simultaneously, fixed frame rate sampling is performed to remove invalid and redundant frames, reducing inference computational power consumption.

[0061] Define the original video sequence: Original video frame rate , The original video frame number, with a value range of... .

[0062] Set frame sampling interval The keyframe sampling index set is defined as: After sampling, the following set of valid keyframes is obtained: .

[0063] Define a set of monitoring scene partitions ,in For public areas This is a private living area.

[0064] If the scene is a public area Directly output the original sampled keyframes No cropping or blurring will be applied; if the scene is a private room. : Launch a YOLOv8 object detection, ROI adaptive cropping, and face mean blurring integrated privacy desensitization pipeline.

[0065] The following is the desensitization process for video images from private living areas:

[0066] Detect a single human target using the YOLOv8 model and output the original human bounding box: Set the adaptive scaling factor for the interaction ROI. The optimal values ​​for this parameter are 1.1, 1.2, and 1.3, which can be adaptively fine-tuned according to the scenario. This is based on the human body bounding box width and height offset. , To allow for adjustments based on specific circumstances, the initial interactive ROI clipping box is generated by expansion: .

[0067] To prevent the cropping boundary from exceeding the image canvas, boundary constraint correction is performed, assuming the original image width is... Height is The revised rules are as follows:

[0068] ;

[0069] After boundary constraints, the effective interactive clipping box is obtained: .

[0070] Interactive sub-images are obtained by cropping from the original keyframes based on effective cropping boxes: .

[0071] In sub-image Internally, perform face detection and obtain a set of face pixel regions. ,satisfy Count the total number of pixels in the face region .

[0072] Taking a single-channel image as an example, for any pixel within the face region Calculate the average pixel value of the face region:

[0073] ;

[0074] For an RGB three-channel image, calculate the mean value of each channel independently. .

[0075] Complete the desensitization and value assignment by traversing all pixels of the face: .

[0076] After desensitization, the facial area is filled with a solid color block, completely eliminating facial features and textures. This preserves the human posture and environmental characteristics within the ROI while meeting privacy compliance requirements. The desensitized area will then be... By stitching the image with a non-privacy background area, a complete desensitized image is output. .

[0077] Step 2: Input the desensitized image into the trained VLM multimodal large model, generate image visual features through a visual encoder, and encode the preset task prompt words into text prompt features through a text embedding layer.

[0078] Understandably, existing nursing behavior recognition solutions suffer from low differentiation between similar actions and a high false alarm rate. In daily nursing scenarios, compliant nursing actions such as wiping the face, helping someone up, tidying bedding, and raising an arm to adjust body position are highly similar in skeletal key point features to non-compliant actions. Traditional solutions rely solely on displacement and velocity thresholds for judgment, which easily generates false alarms, resulting in a large amount of system redundancy and extremely high manual review and maintenance costs.

[0079] This invention leverages the powerful semantic understanding capabilities of the VLM (Visual Language Model) to distinguish different dynamic states. First, a VLM multimodal model is pre-trained, with the weight matrix fine-tuned using LoRA (Local Algorithm for elderly care scenarios). Image semantic encoding is then performed on desensitized keyframes to accurately output multi-dimensional semantic information such as scene entities, human-computer interaction relationships, and environmental states. The VLM model comprises two core modules: a visual encoder and a large language decoder, employing a lightweight deployment scheme of freezing the pre-trained backbone and using low-rank adaptation fine-tuning.

[0080] Define the core weight matrix for model pre-training: ,in Input dimensions for the network, Output dimensions for the network. It is frozen throughout the process and does not participate in gradient updates.

[0081] Introducing the LoRA low-rank fine-tuning mechanism without altering the original pre-trained weights. By inserting two sets of low-rank matrices, efficient parameter fine-tuning is achieved, and the final inference weight calculation formula is derived: In the formula, These are the inference weights for the fine-tuned model. It is a low-rank upward projection matrix; It is a low-rank descending projection matrix; It is a low-rank hyperparameter.

[0082] The input training set for the multimodal large model consists of paired data composed of desensitized images preprocessed in step 1 and corresponding task labels. Specifically, public areas are sampled from the entire screen, preserving complete scene information. Private areas are desensitized images after human detection → ROI dilation and cropping → face mean desensitization → image stitching, preserving human pose, interaction actions, and environmental features, but faces are covered by solid color blocks. For each image, corresponding behavioral descriptions (distance to behavior type) and compliance judgments are manually labeled.

[0083] To achieve effective LoRA fine-tuning, it is recommended to collect no fewer than 5,000 desensitized images processed by S1, covering the following diverse scenarios.

[0084] During the training of a multimodal large model, desensitized images Input the VLM visual encoder to generate image visual feature vectors. Customized task prompts are encoded using a text embedding layer to generate text prompt feature vectors. .

[0085] For image visual feature vectors With text feature vectors The execution dimensions are concatenated and fused to generate multimodal fusion features, which are then fed into a large language decoder for forward inference. During model training, gradients are backpropagated based on the recognition task loss function, and only the LoRA low-rank matrix is ​​updated. The parameters, main intervention training weights The model is frozen indefinitely and iterated until the loss converges, allowing for fine-tuning and adaptation for lightweight deployment on edge devices.

[0086] Step 3: Construct an offline structured regulatory norms knowledge base, which contains vectorized normative text fragments and their semantic tags.

[0087] In one embodiment of the present invention, step 3 involves constructing an offline structured regulatory normative knowledge base, the knowledge base comprising vectorized normative text fragments and their semantic tags, including:

[0088] Collect the pre-set service safety standard texts and assessment rule texts for welfare institutions, classify each standard text according to the pre-set classification set, and obtain the classification label for each standard text;

[0089] Each standard text is sliced ​​according to a set text length threshold, and valid text segments containing descriptions of violations and mandatory constraints are selected to generate a set of standard slices.

[0090] Each specification slice is bound to at least one preset semantic tag and evaluation parameter;

[0091] By using a dedicated embedding model fine-tuned for elderly care scenarios, dense feature vectors are generated for each standardized slice, and a structured offline knowledge base containing classification labels, standardized slices, semantic labels, evaluation parameters, and dense feature vectors is constructed.

[0092] Specifically, this step is based on a structured regulatory knowledge base for the elderly care sector, and mainly consists of the following steps:

[0093] (1) Data source aggregation. Construct a complete set of standardized documents. The data source includes the "Basic Norms for Service Safety of Elderly Care Institutions (GB38600)" and the Civil Affairs Elderly Care Assessment Deduction Standards. Invalid texts such as general principles and interpretations have been removed, and only the normative texts have been retained.

[0094] (2) Primary classification. Pre-set exclusive classification set for welfare institutions. , , , , and To represent different nursing behaviors, each specification document is matched using regular expressions. Assign each document to a unique category and output its category tag. .

[0095] (3) Second-level standardization slicing. Set text length threshold. Long texts are split using periods and semicolons as delimiters. Each specification document is then sliced, and valid segments containing "violation behavior + mandatory constraints" are selected to generate a set of specification slices. Ensure that the length of all slices meets the requirements. , where i represents the index of the canonical document and j represents the index of the canonical slice.

[0096] (4) Three-level labeling. Based on the dictionary of keywords for standardized behavior in building welfare homes. For each standard slice Precise matching generates a unique set of semantic tags. .

[0097] (5) Evaluation parameter binding. Evaluation parameters are bound to each specification slice: These correspond to the type of violation penalty, the fixed penalty value, and the penalty value per unit of time, respectively.

[0098] (6) Domain-specific vectorization and data entry. A dedicated embedding model fine-tuned for the elderly care scenario is used to vectorize each specification slice. Generate a 768-dimensional dense vector: Finally, a structured offline knowledge base is constructed: .

[0099] Step 4: Perform global mean pooling on the image visual features to generate an image semantic retrieval vector, and extract scene semantic keywords corresponding to the image visual features; based on the image semantic retrieval vector and the scene semantic keywords, adopt a two-layer retrieval mechanism of keyword hard matching and vector semantic similarity soft recall to recall matching standard entries from the knowledge base, and calculate the matching confidence score.

[0100] Understandably, in order to completely solve the problem of redundant information from the same source in traditional VLM text retrieval, this algorithm directly generates retrieval vectors based on the visual features of VLM images, without any text transfer links, and relies entirely on the semantics of the image to perform knowledge base retrieval.

[0101] Specifically, extract the image visual feature matrix output by the VLM visual encoder. ,in The length of the feature sequence. This represents the number of single-dimensional feature channels.

[0102] Global mean pooling is performed on the image visual feature matrix to generate a unique retrieval vector:

[0103] ;

[0104] In the formula The image semantic retrieval vector serves as the sole retrieval input for the RAG knowledge base, avoiding the information duplication problem caused by secondary text reasoning.

[0105] After generating image semantic retrieval vectors, this invention features a unique dual-layer retrieval mechanism of [keyword hard matching + vector semantic soft recall], which balances the accuracy of standardization with semantic generalization and is suitable for the service behavior standardization scenarios of welfare homes.

[0106] In one embodiment of the present invention, scene semantic keywords corresponding to the visual features of the image are extracted, and semantic tags in the knowledge base are traversed based on the scene semantic keywords to filter matching standard slices and obtain a hard matching set.

[0107] Calculate the similarity between the image semantic retrieval vector and the dense feature vector of each canonical slice in the knowledge base, and filter the K canonical slices with the highest similarity to obtain the soft recall set;

[0108] The hard matching set and the soft recall set are deduplicated to obtain the final valid specification slice.

[0109] Specifically, the matching method includes the following steps:

[0110] (1) Keyword hard matching retrieval: Extracting the set of scene semantic keywords corresponding to the visual features of the image. For various video scenarios, a pre-set set of corresponding scene semantic keywords is established. During application, the current video scenario can be used to match and retrieve the corresponding set of scene semantic keywords. Based on the scene semantic keyword set corresponding to the image's visual features, all semantic tags in the knowledge base are traversed to accurately match the corresponding standardized slices, forming a hard-match set of keywords.

[0111] .

[0112] (2) Vector similarity soft recall: Calculate the cosine similarity between the image semantic retrieval vector and the dense feature vector of each canonical slice in the knowledge base:

[0113] ;

[0114] Scene-adaptive TopK filtering, sorted by similarity in descending order to obtain the soft recall set: The selection of the K value is adaptively set according to the scenario; for example, in a scenario involving physical punishment, the K value is set to... Scenes of emotional abuse and neglect .

[0115] (3) The hard keyword matching set and the soft recall set are deduplicated and merged using the rule of [mandatory specification priority, unique number deduplication] to obtain the final effective specification slice set:

[0116] .

[0117] Based on the soft recall set results, the current matching confidence score is calculated. In this embodiment of the invention, the maximum similarity within the soft recall set is taken as the matching confidence score.

[0118] .

[0119] Step 5: Concatenate and fuse the image visual features, the text prompt features, and the feature vectors of the recalled standard entries to obtain enhanced fused features;

[0120] Understandably, based on the final set of valid standardized slices obtained through fusion, mean pooling is performed on the dense feature vectors of all valid standardized slices in the set to generate unified knowledge features, represented as: , To effectively standardize the total number of slices.

[0121] The image visual features, text prompt features, and knowledge features are aligned and concatenated to generate the final fused and enhanced features: .

[0122] Step 6: Input the enhanced fusion features into the large language decoder in the VLM multimodal large model for inference to obtain the nursing behavior type.

[0123] Understandably, this involves fusing and enhancing features. The data is fed into the VLM decoder for inference to obtain the nursing behavior type recognition result. This enables the standardization of the model's judgment logic from the feature layer, solving the technical pain point of the inability of traditional visual models to recognize behaviors without limb characteristics.

[0124] Step 7: Based on the matching confidence score, nursing behaviors are classified into different levels, and the welfare institution is evaluated based on the identified nursing behavior type and the duration of continuous behavior.

[0125] It is understood that embodiments of the present invention will match confidence scores. Using dual thresholds specific to a particular scenario as the sole quantitative criterion, reproducible and highly accurate service behavior classification is achieved, completely resolving the issue of misjudgment of highly confusing, homogeneous actions. Specifically, the following steps are included:

[0126] (1) Threshold parameter calibration. Based on real positive and negative samples (compliant and non-compliant samples) from welfare homes, ROC curves were plotted, and empirical parameters were given: low-risk threshold. High-risk threshold .

[0127] (2) Unified formula for three-level classification:

[0128] ;

[0129] (3) Differentiation of highly confusing actions.

[0130] ①Standardized actions: Behaviors should conform to compliant elderly care standards. This was determined to be normal care.

[0131] ②Non-standard actions: This was determined to be a non-standard action.

[0132] (4) Structured output generation. The multimodal large model outputs standardized labels for classification: normal care, suspected, and abnormal care; and outputs matching confidence scores simultaneously. The duration and type of behavior provide structured input for subsequent evaluation and quantification algorithms.

[0133] This invention evaluates the service quality of welfare institutions by outputting nursing behavior types and durations based on a multimodal large model. It establishes a bivariate piecewise mapping evaluation function. The system automatically calculates deductions based on the type and duration of the behavior, providing a basis for civil affairs management. Specifically, it includes the following steps:

[0134] (1): Parameter definition. For behavior type; The duration of the continuous action is measured in seconds (s). A fixed deduction value for behavior 1; Points will be deducted for the duration of behavior 2; The penalty for each behavior is 3 points, and all parameters are taken from the official assessment rules of the Ministry of Civil Affairs.

[0135] (2) Original segmented quantitative evaluation core function:

[0136] ;

[0137] Function rule explanation:

[0138] 1) Type 1: Fixed deduction rule is implemented. A single valid behavior triggers a fixed deduction, which does not accumulate over time;

[0139] 2) Type 2: Logarithmic progressive deduction is used, and the longer the behavior lasts, the higher the deduction rate.

[0140] 3) Type 3: Deductions are made progressively in 10-second increments;

[0141] 4) Low-risk suspected samples: Automatically assigned values Only logs and video clips are retained; no points are automatically deducted, and they are left for manual review and confirmation.

[0142] Finally, a score is obtained. The service quality of welfare institutions is evaluated based on the assessment scores.

[0143] The algorithm of this invention realizes a complete mathematical closed loop from visual feature retrieval to standard determination, behavior classification, and quantitative evaluation. All parameters are calibrable, the process is reproducible, and the rules are implementable.

[0144] See Figure 2 The present invention also provides a welfare institution service evaluation system based on an AI visual multimodal large model, comprising:

[0145] The desensitization processing module 201 is used to collect real-time surveillance video streams from welfare institutions, desensitize videos of private living areas, and obtain desensitized images.

[0146] The feature extraction module 202 is used to input the desensitized image into the trained VLM multimodal large model, generate image visual features through a visual encoder, and encode preset task prompt words into text prompt features through a text embedding layer.

[0147] Module 203 is used to build an offline structured regulatory norms knowledge base, which contains vectorized normative text fragments and their semantic tags;

[0148] The matching module 204 is used to perform global mean pooling on the image visual features to generate an image semantic retrieval vector, and to extract scene semantic keywords corresponding to the image visual features; based on the image semantic retrieval vector and the scene semantic keywords, a two-layer retrieval mechanism of keyword hard matching and vector semantic similarity soft recall is adopted to recall the matching standard entries from the knowledge base and calculate the matching confidence score.

[0149] The fusion module 206 is used to concatenate and fuse the image visual features, the text prompt features, and the feature vectors of the recalled normative entries to obtain enhanced fusion features;

[0150] Inference module 207 is used to input the enhanced fusion features into the large language decoder in the VLM multimodal large model for inference to obtain the nursing behavior type;

[0151] Evaluation module 208 is used to classify nursing behaviors into different levels based on the matching confidence score, and to evaluate welfare institutions based on the identified nursing behavior types and the duration of continuous behavior.

[0152] It is understood that the welfare institution service evaluation system based on AI visual multimodal large model provided by the present invention corresponds to the welfare institution service evaluation method based on AI visual multimodal large model provided in the foregoing embodiments. The relevant technical features of the welfare institution service evaluation system based on AI visual multimodal large model can be referred to the relevant technical features of the welfare institution service evaluation method based on AI visual multimodal large model, and will not be repeated here.

[0153] Please see Figure 3 , Figure 3 This is a schematic diagram illustrating an embodiment of the electronic device provided in this invention. For example... Figure 3 As shown, an embodiment of the present invention provides an electronic device 300, including a memory 310, a processor 320, and a computer program 311 stored in the memory 310 and executable on the processor 320. When the processor 320 executes the computer program 311, it implements the steps of a welfare institution service evaluation method based on an AI visual multimodal large model.

[0154] Please see Figure 4 , Figure 4 This is a schematic diagram illustrating an embodiment of a computer-readable storage medium provided by the present invention. (See diagram below.) Figure 4 As shown, this embodiment provides a computer-readable storage medium 400, on which a computer program 411 is stored. When the computer program 411 is executed by a processor, it implements the steps of a welfare institution service evaluation method based on an AI visual multimodal large model.

[0155] The present invention provides a method and system for evaluating welfare institution services based on an AI-based visual multimodal large model, which has the following beneficial effects:

[0156] (1) Breaking through the limitations of traditional skeletal threshold algorithms, this invention achieves full coverage recognition of implicit behaviors. Based on multimodal semantic understanding and normative knowledge constraints, this invention breaks away from the dependence on limb features and can accurately identify non-standard behaviors, thus completely solving the industry pain point of large-scale missed detections in traditional solutions.

[0157] (2) Balancing privacy compliance and lightweight deployment, this invention adopts partitioned ROI cropping + face mean desensitization preprocessing to shield privacy information in private areas and retain complete scene features in public areas. At the same time, it significantly reduces inference computing power through key frame sampling + LoRA lightweight fine-tuning, solving the technical contradiction of excessive computing power of large full-image models and inability to deploy at the edge, and adapting to the privacy and low-cost deployment needs of welfare homes.

[0158] (3) It has zero-sample generalization ability. Relying on the knowledge base dynamic retrieval enhancement mechanism, the model can adapt to various behaviors based on newly added norms. It does not require a large number of re-labeled samples for training. Its generalization ability is far superior to the traditional fixed threshold and fixed sample training model.

[0159] (4) Achieve a closed-loop system for identification, quantification, evaluation, and early warning. Establish a dual-variable evaluation function for behavior type and duration. It can automatically complete the quality evaluation, graded early warning, and evidence retention of welfare institutions, and realize intelligent, standardized, and quantifiable supervision of welfare institutions.

[0160] It should be noted that the descriptions of each embodiment in the above embodiments have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0161] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0162] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0163] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0164] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1The steps of the function specified in one or more boxes.

[0165] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0166] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A service evaluation method for welfare institutions based on an AI visual multimodal large model, characterized in that, include: Step 1: Real-time acquisition of surveillance video streams from welfare institutions; desensitization processing of video from private living areas to obtain desensitized images; Step 2: Input the desensitized image into the trained VLM multimodal large model, generate image visual features through the visual encoder, and encode the preset task prompt words into text prompt features through the text embedding layer; Step 3: Construct an offline structured regulatory norms knowledge base, which contains vectorized normative text fragments and their semantic tags; Step 4: Perform global mean pooling on the image visual features to generate an image semantic retrieval vector, and extract scene semantic keywords corresponding to the image visual features; based on the image semantic retrieval vector and the scene semantic keywords, adopt a two-layer retrieval mechanism of keyword hard matching and vector semantic similarity soft recall to recall matching standard entries from the knowledge base, and calculate the matching confidence score; Step 5: Concatenate and fuse the image visual features, the text prompt features, and the feature vectors of the recalled standard entries to obtain enhanced fused features; Step 6: Input the enhanced fusion features into the large language decoder in the VLM multimodal large model for inference to obtain the nursing behavior type; Step 7: Based on the matching confidence score, nursing behaviors are classified into different levels, and the welfare institution is evaluated based on the identified nursing behavior type and the duration of continuous behavior.

2. The method according to claim 1, characterized in that, Step 1 involves real-time acquisition of surveillance video streams from the welfare institution, and desensitizing the video footage from private living areas to obtain desensitized images, including: The system collects real-time surveillance video streams from welfare institutions, determines the scene area of ​​the surveillance video stream, and if it is in a public area, no processing is performed on the surveillance video stream; if it is in a private room area, human targets are located through target detection, the region of interest (ROI) is adaptively cropped, and pixel mean desensitization processing is performed on the face area within the ROI, and the desensitized image is stitched together.

3. The method according to claim 2, characterized in that, In step 1, if the location is a private living area, the human target is located through target detection, the Region of Interest (ROI) is adaptively cropped, and pixel mean desensitization processing is performed on the face region within the ROI. The resulting desensitized image is then stitched together. This process includes: For videos of private living areas, human targets within the private living area are detected using a target detection model to obtain human bounding boxes; The human bounding box is expanded according to a preset adaptive magnification factor to generate an initial ROI cropping box. The boundary constraints of the initial ROI cropping box are then corrected by combining the width and height of the video frame image to obtain an effective interactive cropping box. Based on the effective interactive cropping box, a ROI sub-image is cropped from the video frame image. The face region within the ROI sub-image is detected, the average pixel value of each channel in the face region is calculated, and the average pixel value is assigned to all pixels in the face region. The pixels in the face region are desensitized, and the desensitized face region and non-privacy background region are stitched together to obtain a complete desensitized image.

4. The method according to claim 1, characterized in that, The VLM multimodal large model in step 2 employs a lightweight deployment with a frozen pre-trained backbone and low-rank adaptation fine-tuning, including: Freeze the pre-trained core weight matrix W0 of the VLM multimodal large model, introduce low-rank up projection matrix A and low-rank down projection matrix B, and fine-tune the inference weights of the VLM multimodal large model as follows: , where the rank r of B and A is much smaller than the dimension of W0; During the training of the VLM multimodal large model, the gradient is backpropagated based on the loss function of the nursing behavior recognition task. Only the up-projection matrix A and the low-rank down-projection matrix B are updated. The pre-trained core weight matrix W0 is frozen throughout the process. The iteration continues until the loss converges, and the trained VLM multimodal large model is obtained.

5. The method according to claim 1, characterized in that, Step 3 involves constructing an offline structured regulatory norms knowledge base. This knowledge base contains vectorized normative text fragments and their semantic tags, including: Collect the pre-set service safety standard texts and assessment rule texts for welfare institutions, classify each standard text according to the pre-set classification set, and obtain the classification label for each standard text; Each standard text is sliced ​​according to a set text length threshold, and valid text segments containing descriptions of violations and mandatory constraints are selected to generate a set of standard slices. Each specification slice is bound to at least one preset semantic tag and evaluation parameter; For each standardized slice, a dense feature vector is generated, and a structured offline knowledge base containing classification labels, standardized slices, semantic labels, evaluation parameters, and dense feature vectors is constructed.

6. The method according to claim 5, characterized in that, In step 4, based on the image semantic retrieval vector and the scene semantic keywords, a two-layer retrieval mechanism combining keyword hard matching and vector semantic similarity soft recall is used to recall matching standard entries from the knowledge base, including: Extract scene semantic keywords corresponding to the visual features of the image, traverse the semantic tags in the knowledge base based on the scene semantic keywords, filter the matching standard slices, and obtain a hard matching set; Calculate the similarity between the image semantic retrieval vector and the dense feature vector of each canonical slice in the knowledge base, and filter the K canonical slices with the highest similarity to obtain the soft recall set; The hard matching set and the soft recall set are deduplicated to obtain the final valid specification slice.

7. The method according to claim 6, characterized in that, In step 4, the matching confidence score is calculated, including: Calculate the matching confidence score based on the maximum similarity in the soft recall set; Step 5 involves concatenating and fusing the image visual features, the text prompt features, and the feature vectors of the recalled canonical entries to obtain enhanced fusion features, including: Based on the final valid standardized slices, mean pooling is performed on the dense feature vectors of all valid standardized slices to generate knowledge features; The image visual features, the text prompt features, and the knowledge features are concatenated and fused to generate enhanced fusion features.

8. The method according to claim 1, characterized in that, Step 7 involves classifying nursing behaviors into different levels based on the matching confidence score, including: When the matching confidence score is less than the low-risk threshold, it is determined to be normal behavior; When the matching confidence score is greater than or equal to the high-risk threshold, it is determined to be a high-risk behavior; When the matching confidence score is between the low-risk threshold and the high-risk threshold, it is judged as a low-risk suspected behavior and the log is retained for manual review.

9. A service evaluation system for welfare institutions based on an AI visual multimodal large model, characterized in that, include: The desensitization processing module is used to collect real-time surveillance video streams from welfare institutions, desensitize videos from private living areas, and obtain desensitized images. The feature extraction module is used to input the desensitized image into the trained VLM multimodal large model, generate image visual features through a visual encoder, and encode preset task prompt words into text prompt features through a text embedding layer; A construction module is used to build an offline structured regulatory norms knowledge base, which contains vectorized normative text fragments and their semantic tags. The matching module is used to perform global mean pooling on the visual features of the image to generate an image semantic retrieval vector, and to extract scene semantic keywords corresponding to the visual features of the image; based on the image semantic retrieval vector and the scene semantic keywords, a two-layer retrieval mechanism of hard keyword matching and soft recall based on vector semantic similarity is adopted to recall the matching standard entries from the knowledge base and calculate the matching confidence score. The fusion module is used to concatenate and fuse the image visual features, the text prompt features, and the feature vectors of the recalled canonical entries to obtain enhanced fusion features; The inference module is used to input the enhanced fusion features into the large language decoder in the VLM multimodal large model for inference to obtain the nursing behavior type; The evaluation module is used to classify nursing behaviors into different levels based on the matching confidence score, and to evaluate welfare institutions based on the identified nursing behavior types and the duration of continuous behavior.

10. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the steps of the method as claimed in any one of claims 1 to 8.

Images