Ageing-suitable scene evaluation system and method based on multi-modal deep learning, and medium

Abstract

The embodiment of the invention discloses an aging-suitable scene evaluation system and method based on multi-modal deep learning, and a medium. The system collects a target image of an aging-suitable scene to be evaluated; the image preprocessing module preprocesses the target image; the multi-modal feature extraction module extracts scene overall layout features and local object detail features in an aged scene; the layered recognition and judgment module performs scene classification according to the scene overall layout features and the local object detail features, and performs object detection according to a scene classification result; the rule matching analysis module carries out matching analysis according to a pre-created aging-suitable modification rule base; and the transformation suggestion generation module generates a visual aging-suitable scene evaluation report according to an analysis result of the rule matching analysis module. According to the invention, through multi-modal deep learning and spatial relationship reasoning, automatic identification and analysis of the aging-suitable transformation demand are realized, and compared with an existing evaluation scheme, efficiency, precision and cost are significantly improved, and the method is suitable for popularization and application.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of age adaptation, and in particular to an age adaptation scene evaluation system and method based on multi-modal deep learning and a medium. BACKGROUND

[0002] According to the national population census data, the aging degree in China is gradually deepening. With the arrival of an aging society, the elderly either choose home-based care or choose care in a care institution. Whether it is home-based care or care in a care institution, whether the elderly adopt which way of care, whether the living environment is safe, convenient, suitable, and comfortable is the first problem to be solved, and age adaptation becomes an important part of the care environment.

[0003] In age adaptation, age adaptation scene evaluation is used to determine the age adaptation requirements. However, the traditional age adaptation scene evaluation scheme still has the following deficiencies: 1. It relies on manual on-site investigation, and the average time for single-house evaluation is more than half an hour, which is time-consuming and laborious, and the missed detection rate of the transformation requirements due to subjective judgment is as high as more than 20%; 2. It cannot accurately classify similar scenes such as bathrooms and kitchens, and the classification accuracy is less than 70%, and the detection missed detection rate of small targets such as anti-slip mats is as high as 40%; 3. It cannot combine scene semantics and object spatial relationships for joint analysis, resulting in insufficient adaptability of the transformation scheme and lack of joint analysis capability.

[0004] The above problems need to be solved. SUMMARY

[0005] To solve the problems in the related art, the present application provides an age adaptation scene evaluation system and method based on multi-modal deep learning and a medium.

[0006] To achieve the above purpose, the embodiments of the present application adopt the following technical solutions:

[0007] In a first aspect, the embodiments of the present application provide an age adaptation scene evaluation system based on multi-modal deep learning, which comprises:

[0008] An image acquisition module is configured to acquire a target image of an age adaptation scene to be evaluated and output the target image to an image preprocessing module;

[0009] An image preprocessing module is configured to receive the target image of the age adaptation scene to be evaluated, pre-process the target image, and output the pre-processed target image to a multi-modal feature extraction module;

[0010] A multi-modal feature extraction module is configured to extract scene overall layout features and local object detail features in the age adaptation scene based on a double-branch convolutional neural network;

[0011] The hierarchical recognition and judgment module is configured to perform scene classification according to the overall layout features and the local object detail features of the scene, and perform object detection according to the scene classification result.

[0012] The rule matching analysis module is configured to perform matching analysis on the input data of the hierarchical recognition and judgment module according to a pre-created aging modification rule library, and output the analysis result data to the modification suggestion generation module.

[0013] The modification suggestion generation module is configured to generate a visual aging scene evaluation report according to the analysis result of the rule matching analysis module.

[0014] As an optional implementation, the image preprocessing module is specifically configured to:

[0015] receive a target image of the aging scene to be evaluated, and perform processing including but not limited to size standardization, illumination enhancement, and noise suppression on the target image.

[0016] As an optional implementation, the multi-modal feature extraction module is specifically configured to:

[0017] extract overall layout features of the aging scene based on a global branch convolutional neural network, and extract local object detail features of the aging scene based on a local branch convolutional neural network.

[0018] As an optional implementation, the hierarchical recognition and judgment module includes a scene classification module and an object detection module; the scene classification module is configured to fine-tune a pre-trained model, introduce scene semantic labels, and perform scene classification; and the object detection module is configured to call a special detection model for object detection according to the scene classification result output by the scene classification module.

[0019] As an optional implementation, the rule matching analysis module is specifically configured to:

[0020] perform spatial relationship analysis of the aging scene according to a pre-created aging modification rule library, and output the analysis result to the modification suggestion generation module.

[0021] As an optional implementation, the modification suggestion generation module supports mobile terminal real-time analysis and automatically generates a visual aging scene evaluation report.

[0022] As an optional implementation, the hierarchical recognition and judgment module is specifically configured to: synchronize optimization of the scene classification and the object detection tasks through a multi-task learning framework, and model the spatial relationship between objects through a graph neural network (GNN).

[0023] In a second aspect, an embodiment of the present application provides an old-age adaptation scene evaluation method based on multi-modal deep learning, which comprises the following steps:

[0024] An image acquisition module acquires a target image of an old-age adaptation scene to be evaluated and outputs the target image to an image preprocessing module.

[0025] The image preprocessing module receives the target image of the old-age adaptation scene to be evaluated, pre-processes the target image, and outputs the pre-processed target image to a multi-modal feature extraction module.

[0026] The multi-modal feature extraction module extracts overall layout features and local object detail features of the old-age adaptation scene based on a double-branch convolutional neural network.

[0027] A hierarchical recognition and judgment module performs scene classification according to the overall layout features and the local object detail features, and performs object detection according to the scene classification result.

[0028] A rule matching analysis module performs matching analysis on the data input by the hierarchical recognition and judgment module according to a pre-created old-age adaptation rule library, and outputs analysis result data to a reconstruction suggestion generation module.

[0029] The reconstruction suggestion generation module generates a visual old-age adaptation scene evaluation report according to the analysis result of the rule matching analysis module.

[0030] As an optional implementation, the image preprocessing module receives the target image of the old-age adaptation scene to be evaluated, pre-processes the target image, and specifically comprises the following steps:

[0031] The target image of the old-age adaptation scene to be evaluated is received, and the target image is pre-processed, including but not limited to size normalization, illumination enhancement, and noise suppression.

[0032] In a third aspect, an embodiment of the present application provides a computer readable storage medium, wherein the computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by a processor to implement the old-age adaptation scene evaluation method based on multi-modal deep learning provided in the second aspect.

[0033] The technical scheme provided by the embodiment of the present application collects a target image of an aging-adaptable scene to be evaluated by an image acquisition module and outputs the target image to an image preprocessing module; the image preprocessing module receives the target image of the aging-adaptable scene to be evaluated, pre-processes the target image, and outputs the pre-processed target image to a multi-modal feature extraction module; the multi-modal feature extraction module extracts overall layout features and local object detail features of the aging-adaptable scene based on a double-branch convolutional neural network; a hierarchical recognition and judgment module performs scene classification according to the overall layout features and the local object detail features, and performs object detection according to the scene classification result; a rule matching and analysis module performs matching analysis on the data input by the hierarchical recognition and judgment module according to a pre-created aging-adaptable reconstruction rule library, and outputs analysis result data to a reconstruction suggestion generation module; and the reconstruction suggestion generation module generates a visual aging-adaptable scene evaluation report according to the analysis result of the rule matching and analysis module. The technical scheme provided by the embodiment of the present application realizes automatic recognition and analysis of aging-adaptable reconstruction requirements through multi-modal deep learning and spatial relationship reasoning, and significantly improves efficiency, accuracy and cost compared with existing evaluation schemes, and is suitable for popularization and application. BRIEF DESCRIPTION OF DRAWINGS

[0034] In order to more clearly illustrate and understand the technical solutions in the embodiments of the present application, the drawings needed to be used in the background of the present application and the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of the contents of the embodiments of the present application and these drawings.

[0035] Figure 1 The aging-adaptable scene evaluation system structure block diagram based on multi-modal deep learning provided by the embodiment of the present application is provided.

[0036] Figure 2 The aging-adaptable scene evaluation method flowchart based on multi-modal deep learning provided by the embodiment of the present application is provided. DETAILED DESCRIPTION

[0037] In order to make the technical problems solved by the present application, the technical solutions adopted and the technical effects achieved more clear, the technical solutions of the embodiments of the present application will be further described in detail below with reference to the drawings. Obviously, the described embodiments are only some of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0038] Embodiment one

[0039] Please refer to Figure 1 The, Figure 1A structural block diagram of an old-age adaptation scene evaluation system based on multi-modal deep learning is provided for an embodiment of the present application. As shown in the figure, the old-age adaptation scene evaluation system 100 based on multi-modal deep learning in the present embodiment comprises:

[0040] An image acquisition module 101 is configured to acquire a target image of an old-age adaptation scene to be evaluated and output to an image preprocessing module 102.

[0041] The image preprocessing module 102 is configured to receive the target image of the old-age adaptation scene to be evaluated, pre-process the target image, and output to a multi-modal feature extraction module 103.

[0042] The multi-modal feature extraction module 103 is configured to extract overall layout features and local object detail features in the old-age adaptation scene based on a double-branch convolutional neural network.

[0043] A hierarchical recognition and judgment module 104 is configured to perform scene classification according to the overall layout features and local object detail features, and perform object detection according to the scene classification result.

[0044] A rule matching analysis module 105 is configured to perform matching analysis on the input data of the hierarchical recognition and judgment module 104 according to a pre-created old-age adaptation rule library, and output the analysis result data to a renovation suggestion generation module 106.

[0045] The renovation suggestion generation module 106 is configured to generate a visual old-age adaptation scene evaluation report according to the analysis result of the rule matching analysis module 105.

[0046] For example, the image preprocessing module 102 is specifically configured to receive the target image of the old-age adaptation scene to be evaluated, and perform processing including but not limited to size normalization, illumination enhancement, and noise suppression on the target image.

[0047] For example, the multi-modal feature extraction module 103 is specifically configured to extract overall layout features in the old-age adaptation scene based on a global branch convolutional neural network, and extract local object detail features in the old-age adaptation scene based on a local branch convolutional neural network.

[0048] For example, the hierarchical recognition and judgment module 104 comprises a scene classification module 1041 and an object detection module 1042. The scene classification module 1041 is configured to fine-tune based on a pre-trained model, introduce scene semantic labels, and perform scene classification. The object detection module 1042 is configured to call a special detection model for object detection according to the scene classification result output by the scene classification module.

[0049] Exemplarily, the rule matching analysis module 105 is specifically configured to: perform spatial relationship analysis on the aging adaptation scene according to a pre-created aging adaptation rule library, and output an analysis result to the reconstruction suggestion generation module 106.

[0050] Exemplarily, the reconstruction suggestion generation module 106 supports mobile terminal real-time analysis, and automatically generates a visual aging adaptation scene evaluation report.

[0051] Exemplarily, the hierarchical identification and judgment module 104 is specifically configured to: synchronize optimization of scene classification and object detection tasks through a multi-task learning framework, and model spatial relationships between objects through a graph neural network (GNN).

[0052] The aging adaptation scene evaluation system 100 based on multi-modal deep learning proposed in this embodiment realizes automatic identification and analysis of aging adaptation reconstruction needs through multi-modal deep learning and spatial relationship reasoning, and significantly improves efficiency, accuracy and cost compared with existing evaluation schemes.

[0053] Embodiment Two

[0054] The aging adaptation scene evaluation system based on multi-modal deep learning in this embodiment specifically includes:

[0055] An image acquisition module configured to acquire a target image of an aging adaptation scene to be evaluated and output the target image to an image preprocessing module;

[0056] An image preprocessing module configured to receive the target image of the aging adaptation scene to be evaluated, perform preprocessing on the target image, and output the preprocessed target image to a multi-modal feature extraction module;

[0057] A multi-modal feature extraction module configured to extract overall layout features and local object detail features of the aging adaptation scene based on a double-branch convolutional neural network;

[0058] A hierarchical identification and judgment module configured to perform scene classification according to the overall layout features and the local object detail features, and perform object detection according to the scene classification result;

[0059] A rule matching analysis module configured to perform matching analysis on data input by the hierarchical identification and judgment module according to a pre-created aging adaptation rule library, and output analysis result data to a reconstruction suggestion generation module;

[0060] A reconstruction suggestion generation module configured to generate a visual aging adaptation scene evaluation report according to the analysis result of the rule matching analysis module.

[0061] Exemplarily, the image preprocessing module is specifically configured to: receive a target image of the to-be-evaluated aging-adapted scene, and perform processing including but not limited to size normalization, illumination enhancement, and noise suppression on the target image.

[0062] Specifically, in this embodiment, the size normalization includes: using a bilinear interpolation algorithm to uniformly adjust an input image to a resolution of 1024x768. In this way, the model input requirement can be adapted, and the performance of the model caused by inconsistent image sizes can be avoided. The illumination enhancement includes: CLAHE histogram equalization, with a contrast limit of 0.01, a block size of 8x8, and enhanced detail visibility in dark light areas. Adaptive gamma correction: dynamically adjusting the gamma value according to the image brightness to balance the contrast of over-bright and over-dark areas. In this way, the image quality under complex lighting conditions (such as shadows and backlight) can be improved, and the false detection caused by uneven lighting can be reduced. The noise suppression includes: using a multi-scale Gaussian filter (σ=1.5) to perform multi-level smoothing processing on the image. In this way, random noise (such as camera shake and low-light noise points) in the image can be reduced, and the robustness of subsequent feature extraction can be ensured.

[0063] Exemplarily, the multi-modal feature extraction module is specifically configured to: extract overall layout features of the aging-adapted scene based on a global branch convolutional neural network, and extract local object detail features of the aging-adapted scene based on a local branch convolutional neural network.

[0064] Specifically, in this embodiment, the multi-modal feature extraction module adopts a double-branch convolutional neural network architecture; for the global branch convolutional neural network: scene overall layout features (such as room structure and furniture distribution) are extracted based on ResNet-50; for the local branch convolutional neural network: local object detail features (such as the positions and sizes of key objects such as toilets and handrails) are extracted based on YOLOv7. In this embodiment, the multi-modal feature extraction module is also used for feature fusion, specifically, the global and local features are dynamically weighted through an attention mechanism (SE Block) to improve the model's ability to distinguish complex scenes.

[0065] Exemplarily, the hierarchical recognition and judgment module includes a scene classification module and an object detection module; the scene classification module is configured to fine-tune a pre-trained model and introduce scene semantic labels to perform scene classification; and the object detection module is configured to call a special detection model for object detection according to a scene classification result output by the scene classification module.

[0066] Specifically, in the present embodiment, the scene classification module adopts a transfer learning strategy, fine-tunes a pre-trained model (such as EfficientNet-B4), and introduces scene semantic labels (such as 'wet area in bathroom' and 'operation table in kitchen') to improve classification accuracy. Output example: scene type probability distribution (such as bathroom 92.3%, kitchen 7.2%, and bedroom 0.5%). Specifically, in the present embodiment, the object detection module adopts a dynamic model activation mechanism, and calls a special detection model according to the scene classification result output by the scene classification module. For example, in a bathroom scene, a bathroom-specific detection model (labels: toilet, handrail, and non-slip mat) is activated; in a kitchen scene, a kitchen-specific detection model (labels: cooking table, extractor hood, and non-slip mat) is activated. In the present embodiment, the detection box is generated: redundant detection boxes are eliminated through non-maximum suppression (NMS) to ensure that the positioning accuracy of key objects (such as a toilet) is ≥95%.

[0067] Specifically, in the present embodiment, the hierarchical recognition and judgment module jointly trains a model, synchronously optimizes the scene classification and object detection tasks through a multi-task learning framework (Multi-Task Learning), and models the spatial relationship between objects through a graph neural network (GNN).

[0068] For example, the rule matching analysis module is specifically configured to: perform spatial relationship analysis on the age-friendly scene according to a pre-created age-friendly renovation rule library, and output the analysis result to the renovation suggestion generation module.

[0069] Specifically, in the present embodiment, the age-friendly renovation rule library takes the following examples: 1. Wall-toilet distance: ≥40 cm (according to GB 50763-2012 'Barrier-free Design Specification'). 2. Non-slip mat size: area ≥0.6m 2 , covering the wet and slippery area of the floor. 3. Handrail installation height: 80-90 cm from the ground, length ≥60 cm. For spatial relationship analysis, geometric calculation is adopted: the distance between two points is verified through the formula to determine whether the distance meets the rules; semantic reasoning is adopted: the scene semantics (such as 'whether the toilet is close to the water source') are combined to determine the applicability of the rules.

[0070] For example, the renovation suggestion generation module supports real-time analysis on mobile terminals and automatically generates a visual age-friendly scene evaluation report.

[0071] Specifically, in the embodiment, the transformation suggestion generation module adopts lightweight model compression, adopts knowledge distillation and channel pruning technology, compresses the model volume to 30% of the original model, supports mobile terminal real-time analysis (single image processing time ≤1.2s); Full-process automation: The technical scheme of the application only needs 5 minutes from shooting to report generation, which is 600% more efficient than manual evaluation. Small object detection accuracy improvement: Through the multi-scale feature pyramid network (FPN), the small object detection mAP@0.5 reaches 76.3% (improving 18.6% compared with the baseline model). The technical scheme of the application lightweight model: using TensorRT acceleration and INT8 quantization technology, the model inference speed is improved by 3 times; End-to-end automation: realize'shooting-analysis-report' full-process automation, without manual intervention.

[0072] In practical application, it is found through test that the old-age adaptation scene evaluation system based on multi-modal deep learning provided in the embodiment has obvious effect improvement, scene classification accuracy: ≥92.5% (improving 18.6% compared with the baseline model); Small object detection mAP@0.5: 76.3%; Single image processing time: ≤1.2s; Efficiency improvement: single household evaluation time is shortened from 30 minutes to 5 minutes, and the efficiency is improved by 83%; Cost reduction: reduce labor cost through automatic process, and single household evaluation cost is reduced by 65%; Demand identification completeness: the transformation demand identification completeness is improved to 98%.

[0073] The old-age adaptation scene evaluation system based on multi-modal deep learning provided in the embodiment solves the subjectivity and low efficiency problems of traditional manual evaluation by fusing scene classification, fine-grained target detection and spatial relationship reasoning, and is suitable for home environment transformation, intelligent upgrading of old-age care institutions and other scenes. The old-age adaptation scene evaluation system based on multi-modal deep learning provided in the embodiment realizes automatic identification and analysis of old-age adaptation transformation demand through multi-modal deep learning and spatial relationship reasoning, and is significantly improved in efficiency, accuracy and cost compared with existing evaluation schemes.

[0074] Embodiment three

[0075] Please refer to Figure 2 The, Figure 2 The old-age adaptation scene evaluation method based on multi-modal deep learning provided in the embodiment is shown in the flowchart. As shown in the figure, the old-age adaptation scene evaluation method based on multi-modal deep learning 200 in the embodiment includes the following steps:

[0076] S201. The image acquisition module acquires the target image of the old-age adaptation scene to be evaluated, and outputs it to the image preprocessing module;

[0077] S202. The image preprocessing module receives a target image of the aging scene to be evaluated, and outputs the target image after preprocessing to the multi-modal feature extraction module;

[0078] S203. The multi-modal feature extraction module extracts scene overall layout features and local object detail features in the aging scene based on a double-branch convolutional neural network.

[0079] S204. The hierarchical recognition and judgment module performs scene classification according to the scene overall layout features and the local object detail features, and performs object detection according to the scene classification result.

[0080] S205. The rule matching and analysis module performs matching analysis on the input data of the hierarchical recognition and judgment module according to a pre-created aging modification rule library, and outputs the analysis result data to the modification suggestion generation module.

[0081] S206. The modification suggestion generation module generates a visual aging scene evaluation report according to the analysis result of the rule matching and analysis module.

[0082] In this embodiment, the image preprocessing module receives a target image of the aging scene to be evaluated, and performs preprocessing on the target image, including but not limited to size normalization, illumination enhancement, and noise suppression.

[0083] Specifically, in this embodiment, the size normalization includes: using a bilinear interpolation algorithm to uniformly adjust the input image to a resolution of 1024x768. In this way, the model input requirements can be adapted, and the performance of the model caused by inconsistent image sizes can be avoided. The illumination enhancement includes: CLAHE histogram equalization, contrast limit of 0.01, block size of 8x8, and enhancement of detail visibility in dark light areas. Adaptive gamma correction: dynamically adjusting the gamma value according to the image brightness to balance the contrast of over-bright and over-dark areas. In this way, the image quality under complex lighting conditions (such as shadows and backlight) can be improved, and the false detection caused by uneven lighting can be reduced. The noise suppression includes: using a multi-scale Gaussian filter (σ=1.5) to perform multi-level smoothing processing on the image. In this way, random noise (such as camera shake and low-light noise points) in the image can be reduced, and the robustness of subsequent feature extraction can be ensured.

[0084] In this embodiment, the multi-modal feature extraction module extracts scene overall layout features in the aging scene based on a global branch convolutional neural network, and extracts local object detail features in the aging scene based on a local branch convolutional neural network.

[0085] In this embodiment, the multi-modal feature extraction module adopts a double-branch convolutional neural network architecture; for the global branch convolutional neural network: based on ResNet-50 to extract the overall layout features of the scene (such as room structure, furniture distribution); for the local branch convolutional neural network: based on YOLOv7 to extract local object detail features (such as the position and size of key objects such as toilets, handrails, etc.). In this embodiment, the multi-modal feature extraction module is also used for feature fusion, specifically, the global and local features are dynamically weighted through the attention mechanism (SE Block) to improve the model's ability to distinguish complex scenes.

[0086] In this embodiment, the hierarchical recognition and judgment module includes a scene classification module and an object detection module; the scene classification module is fine-tuned based on a pre-trained model and introduces scene semantic labels for scene classification; the object detection module calls a special detection model for object detection according to the scene classification result output by the scene classification module.

[0087] Specifically, in this embodiment, the scene classification module adopts a transfer learning strategy, fine-tunes a pre-trained model (such as EfficientNet-B4), and introduces scene semantic labels (such as 'wet area in bathroom', 'operation table in kitchen') to improve classification accuracy. Output example: scene type probability distribution (such as bathroom 92.3%, kitchen 7.2%, bedroom 0.5%). Specifically, in this embodiment, the object detection module adopts a dynamic model activation mechanism, calls a special detection model according to the scene classification result output by the scene classification module, for example, bathroom scene: activate the bathroom special detection model (labels: toilet, handrail, non-slip mat); kitchen scene: activate the kitchen special detection model (labels: stove, extractor hood, non-slip mat); in this embodiment, the detection box is generated: through non-maximum suppression (NMS) to eliminate redundant detection boxes, to ensure that the positioning accuracy of key objects (such as toilets) is ≥95%.

[0088] Specifically, in this embodiment, the hierarchical recognition and judgment module jointly trains the model, synchronously optimizes the scene classification and object detection tasks through a multi-task learning framework (Multi-Task Learning), and models the spatial relationship between objects through a graph neural network (Graph Neural Network, GNN).

[0089] For example, the rule matching analysis module performs spatial relationship analysis of the age-friendly scene according to the pre-created age-friendly modification rule library, and outputs the analysis result to the modification suggestion generation module.

[0090] Specifically, in the present embodiment, the old-age adaptation rule base takes the following examples: 1. Wall and toilet spacing: ≥40 cm (according to GB 50763-2012 “Barrier-free Design Specification”). 2. Non-slip mat size: area ≥0.6m 2 , covering the wet and slippery area of the ground. 3. Handrail installation height: 80-90 cm from the ground, length ≥60 cm. For spatial relationship analysis, geometric calculation is used: verify whether the distance between two points meets the rules through the distance formula; semantic reasoning is used: combined with scene semantics (such as 'toilet near water source') to judge rule applicability.

[0091] For example, the renovation suggestion generation module supports real-time analysis on mobile terminals and automatically generates a visual old-age adaptation scene evaluation report.

[0092] Specifically, in the present embodiment, the renovation suggestion generation module uses lightweight model compression, adopts knowledge distillation and channel pruning technology, and compresses the model volume to 30% of the original model, supporting real-time analysis on mobile terminals (single image processing time ≤1.2s); full-process automation: the technical solution of the present invention only needs 5 minutes from shooting to report generation, which is 600% more efficient than manual evaluation. Small object detection accuracy improvement: through the multi-scale feature pyramid network (FPN), the small object detection mAP@0.5 reaches 76.3% (improving 18.6% compared with the baseline model). The lightweight model of the technical solution of the present invention: using TensorRT acceleration and INT8 quantization technology, the model inference speed is improved by 3 times; end-to-end automation: realizing'shooting-analysis-report' full-process automation without human intervention.

[0093] In practical applications, it has been found through experiments that the old-age adaptation scene evaluation method based on multi-modal deep learning proposed in the present embodiment has obvious effect improvement, scene classification accuracy: ≥92.5% (improving 18.6% compared with the baseline model); small object detection mAP@0.5: 76.3%; single image processing time: ≤1.2s; efficiency improvement: single household evaluation time is shortened from 30 minutes to 5 minutes, efficiency is improved by 83%; cost reduction: through the automatic process, the labor cost is reduced, and the single household evaluation cost is reduced by 65%; demand identification completeness: the renovation demand identification completeness is improved to 98%.

[0094] The multi-modal deep learning-based aging adaptation scene evaluation method 200 proposed in this embodiment solves the subjectivity and inefficiency of traditional manual evaluation by fusing scene classification, fine-grained object detection and spatial relationship reasoning, and is suitable for scenes such as home environment renovation and intelligent upgrading of elderly care institutions. The multi-modal deep learning-based aging adaptation scene evaluation method 200 proposed in this embodiment realizes the automatic recognition and analysis of aging adaptation requirements through multi-modal deep learning and spatial relationship reasoning, and significantly improves the efficiency, accuracy and cost compared with existing evaluation schemes.

[0095] Embodiment Four

[0096] The embodiment provides a computer readable storage medium, and the computer readable storage medium stores computer execution instructions. The computer execution instructions are executed by a processor to implement the multi-modal deep learning-based aging adaptation scene evaluation method 200 provided in the above embodiment one.

[0097] It should be noted that the readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0098] Note that the above are only preferred embodiments of the present application and the principles of technology used. Those skilled in the art will understand that the present application is not limited to the specific embodiments described herein, and those skilled in the art can make various obvious changes, readjustments and substitutions without departing from the scope of the present application. Therefore, although the present application has been described in more detail through the above embodiments, the present application is not limited to the above embodiments, and can include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the appended claims.

Claims

1. A multimodal deep learning-based aging-friendly scenario assessment system, characterized by: include: An image acquisition module is used to acquire the target image of the aging-friendly scene to be evaluated and output it to the image preprocessing module; An image preprocessing module is used to receive the target image of the aging-friendly scene to be evaluated, preprocess the target image, and then output it to the multimodal feature extraction module; A multimodal feature extraction module, which is used to extract the overall layout features and local object detail features in aging-friendly scenes based on a dual-branch convolutional neural network; A hierarchical recognition and judgment module is used to classify scenes based on the overall layout characteristics of the scene and the detailed characteristics of local objects, and to perform object detection based on the scene classification results; A rule matching analysis module is used to perform matching analysis on the input data of the hierarchical identification and judgment module according to a pre-created aging-friendly transformation rule library, and output the analysis result data to the transformation suggestion generation module; The transformation suggestion generation module is used to generate a visual aging-friendly scenario assessment report based on the analysis results of the rule matching analysis module.

2. The aging-friendly scenario assessment system based on multimodal deep learning according to claim 1 is characterized in that: The image preprocessing module is specifically used for: The target image of the aging-friendly scene to be evaluated is received, and the target image is processed including but not limited to size normalization, illumination enhancement, and noise suppression.

3. The aging-friendly scenario assessment system based on multimodal deep learning according to claim 1 is characterized in that: The multimodal feature extraction module is specifically used for: Based on the global branch convolutional neural network, the overall layout features of the scene in the aging-friendly scene are extracted; based on the local branch convolutional neural network, the detailed features of local objects in the aging-friendly scene are extracted.

4. The aging-friendly scenario assessment system based on multimodal deep learning according to claim 3 is characterized in that: The hierarchical recognition and judgment module includes a scene classification module and an object detection module; The scene classification module is used to perform fine-tuning based on the pre-trained model and introduce scene semantic labels to perform scene classification; The object detection module is used to call a dedicated detection model to perform object detection according to the scene classification result output by the scene classification module.

5. The aging-friendly scene assessment system based on multimodal deep learning according to claim 1 is characterized in that: The rule matching analysis module is specifically used to: Based on the pre-created aging-friendly renovation rule library, spatial relationship analysis of aging-friendly scenarios is performed, and the analysis results are output to the renovation suggestion generation module.

6. The aging-friendly scene assessment system based on multimodal deep learning according to claim 1 is characterized in that: The transformation suggestion generation module supports real-time analysis on mobile terminals and automatically generates a visual aging-friendly scenario assessment report.

7. The aging-friendly scene assessment system based on multimodal deep learning according to any one of claims 1 to 6, characterized in that: The hierarchical recognition and judgment module is specifically used to: synchronously optimize scene classification and object detection tasks through a multi-task learning framework, and model the spatial relationship between objects through a graph neural network.

8. A method for evaluating aging-friendly scenarios based on multimodal deep learning, characterized in that: The method adopts the multimodal deep learning aging-friendly scene assessment system described in claim 1, comprising: The image acquisition module acquires the target image of the aging-friendly scene to be evaluated and outputs it to the image preprocessing module; The image preprocessing module receives the target image of the aging-friendly scene to be evaluated, preprocesses the target image, and then outputs it to the multimodal feature extraction module; The multimodal feature extraction module extracts the overall layout features and local object detail features of the aging-friendly scene based on a dual-branch convolutional neural network; The hierarchical recognition and judgment module classifies the scene according to the overall layout characteristics of the scene and the detailed characteristics of the local objects, and performs object detection based on the scene classification results; The rule matching analysis module performs matching analysis on the input data of the hierarchical identification and judgment module according to the pre-created aging-friendly transformation rule library, and outputs the analysis result data to the transformation suggestion generation module; The transformation suggestion generation module generates a visual aging-friendly scenario assessment report based on the analysis results of the rule matching analysis module.

9. The aging-friendly scenario assessment method based on multimodal deep learning according to claim 8 is characterized in that: The image preprocessing module receives the target image of the aging-friendly scene to be evaluated and preprocesses the target image, specifically including: The target image of the aging-friendly scene to be evaluated is received, and the target image is processed including but not limited to size normalization, illumination enhancement, and noise suppression.

10. A computer-readable storage medium storing computer-executable instructions, characterized in that: When the computer execution instructions are executed by the processor, they are used to implement the multimodal deep learning aging-friendly scenario assessment method described in claim 8.

Images