RGB-D-based old-age care community intelligent mapping method and system

Abstract

The invention provides an intelligent mapping method and system for an old-age care community based on RGB-D. The method relates to the technical field of computer vision and robots, and comprises the following steps: acquiring an environment image of the old-age care community; processing the environment image to generate visual feature points and depth information; constructing an initial three-dimensional point cloud map according to the visual feature points and the depth information; pixel-level segmentation is carried out on the RGB color image to obtain a plurality of types of pension devices; associating the initial three-dimensional point cloud map with the position of the pension equipment; based on the association result, combining with the structured geometric constraint, and performing back-end optimization on the initial three-dimensional point cloud map through a factor graph model to obtain an optimized three-dimensional point cloud map; storing the optimized three-dimensional point cloud map and semantic information of the old-age care equipment associated with the optimized three-dimensional point cloud map into a structured semantic map database; and under the condition that the environment of the pension community is changed, performing incremental updating on the structured semantic map database.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision and robot technology, and particularly relates to an intelligent mapping method and system for a pension community based on RGB-D. BACKGROUND

[0002] With the aggravation of the social population aging in China, the pension community plays an increasingly key role in the daily life, nursing and security of the elderly. In order to improve the service intelligence and management efficiency of the pension community, it is urgent to build a high-precision indoor map with spatial perception, semantic understanding and scene recognition capability to realize precise positioning, path navigation, auxiliary decision-making and emergency response of pension facilities.

[0003] The pension community has the characteristics of complex spatial structure, various types of facilities, and frequent service interaction, and its mapping system needs to face the real challenges of unstable network, limited hardware resources, and high data redundancy. Under this condition, the RGB-D perception technology by fusing visual and depth information becomes an important means to build a three-dimensional semantic map, which can provide rich environmental details and structural information for the pension scene.

[0004] However, the current laser SLAM or traditional visual SLAM generates a point cloud map or a grid map at the geometric level, which is a "silent" map and cannot understand the specific meaning of objects in the environment (for example, this is a bed, that is a handrail). The lack of such semantic information greatly limits the application of SLAM technology in intelligent pension and other scenes that need to interact with the environment semantically. SUMMARY

[0005] In view of the above deficiencies of the prior art, the purpose of the embodiments of the present application is to provide an intelligent mapping method for a pension community based on RGB-D, which can solve the technical problem that the current laser SLAM or traditional visual SLAM generates a point cloud map or a grid map at the geometric level, which is a "silent" map and cannot understand the specific meaning of objects in the environment (for example, this is a bed, that is a handrail). The lack of such semantic information greatly limits the application of SLAM technology in intelligent pension and other scenes that need to interact with the environment semantically.

[0006] The first aspect of the embodiments of the present application provides an intelligent mapping method for a pension community based on RGB-D, comprising:

[0007] S1: collecting environment images of the pension community, wherein the environment images include RGB color images and depth images;

[0008] S2: processing the environment images to generate visual feature points and depth information;

[0009] S3: According to the visual feature points and the depth information, an initial three-dimensional point cloud map is constructed through a feature matching and pose estimation algorithm based on depth constraint;

[0010] S4: Through a pre-trained semantic segmentation neural network model, pixel-level segmentation is performed on the RGB color image to obtain a plurality of categories of elderly care devices;

[0011] S5: The initial three-dimensional point cloud map and the positions of the elderly care devices are associated in combination with the elderly care devices of each category and the labels corresponding to the elderly care devices;

[0012] S6: Based on the association result, the initial three-dimensional point cloud map is optimized through a factor graph model in combination with structured geometric constraints to obtain an optimized three-dimensional point cloud map;

[0013] S7: The optimized three-dimensional point cloud map and the semantic information of the elderly care devices associated with the optimized three-dimensional point cloud map are stored in a structured semantic map database;

[0014] S8: In the case that the environment of the elderly care community changes, the structured semantic map database is incrementally updated through a revisit detection mechanism.

[0015] The second aspect of the embodiment of the present application provides an RGB-D based intelligent mapping system for an elderly care community, which comprises a processor and a memory.

[0016] The memory stores programs or instructions executable on the processor, and the programs or instructions are executed by the processor to implement the steps of the RGB-D based intelligent mapping method for an elderly care community according to the first aspect.

[0017] The third aspect of the embodiment of the present application provides a readable storage medium, and the readable storage medium stores programs or instructions, and the programs or instructions are executed by the processor to implement the steps of the RGB-D based intelligent mapping method for an elderly care community according to the first aspect.

[0018] The technical scheme provided by the embodiment of the present application has at least the following beneficial effects:

[0019] In the embodiment of the present application, the environment color image and the depth map are collected by the RGB-D camera, the initial three-dimensional map is quickly generated in combination with the depth constraint feature point matching, and the objects such as beds, handrails, corridors and stairs are labeled by using the semantic segmentation, so that the semanticization of the map is realized. Further, the specific geometric constraints of the elderly care community (such as the corridor straight line constraint and the furniture plane alignment constraint) are introduced to optimize the map precision, generate a maintainable semantic map database, quickly construct a high-precision and semantic-rich community map, provide basic data support for fall detection, intelligent navigation and community management, and significantly improve the efficiency and reliability of the intelligent application of the elderly care community. Attached Figure Description

[0020] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts. Obviously, the drawings described below are merely some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without any creative effort.

[0021] Figure 1 This is a flowchart illustrating an intelligent mapping method for elderly care communities based on RGB-D, provided in an embodiment of the present invention.

[0022] Figure 2 This is a schematic diagram of the structure of an intelligent mapping system for elderly care communities based on RGB-D, provided in an embodiment of the present invention. Detailed Implementation

[0023] To enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. It should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0024] The following description, in conjunction with the accompanying drawings, details the RGB-D-based intelligent mapping method for elderly care communities provided by the present invention through specific embodiments and application scenarios.

[0025] Reference manual attached Figure 1 The diagram illustrates a flowchart of an RGB-D-based intelligent mapping method for elderly care communities provided by an embodiment of the present invention.

[0026] This invention provides an intelligent mapping method for elderly care communities based on RGB-D, which may include the following steps:

[0027] S1: Collect environmental images of the senior living community, including RGB color images and depth images.

[0028] Among them, RGB color images are color images composed of three channels: red, green, and blue, and are mainly used for texture and semantic information extraction.

[0029] Among them, the depth image is used to record the physical distance of each pixel to the camera, usually obtained by a depth camera or RGB-D camera, providing spatial geometric structure information.

[0030] In a possible implementation, S1 is further followed by:

[0031] The depth image is preprocessed by denoising and filtering.

[0032] S2: processing the environment image to generate visual feature points and depth information.

[0033] Among them, the visual feature point refers to a key point in the image with local uniqueness, easy to identify and match, such as corner point, edge point, and common algorithms include ORB, SIFT, etc. Depth information refers to three-dimensional spatial distance data provided by a depth image, used to reconstruct the geometric shape of a scene.

[0034] It should be noted that by extracting visual feature points from the environment image and obtaining corresponding depth information, the system can restore the three-dimensional spatial structure from the two-dimensional image. Visual feature points have good repeatability and distinguishability, and are the core elements of multi-frame image matching, camera motion estimation and map construction. The introduction of depth information further enhances the spatial positioning ability of feature points, so that each key point in the image can be projected as a spatial point with clear three-dimensional coordinates. Compared with traditional methods that rely only on RGB images, the fusion of depth information significantly improves the matching accuracy and map structure stability, especially in typical elderly community scenes with uneven lighting and sparse texture, showing stronger robustness. In addition, this processing procedure provides high-quality data support for subsequent pose estimation, point cloud construction and semantic labeling, and is a basic link for realizing an intelligent mapping system with high precision and strong adaptability.

[0035] S3: according to the visual feature points and the depth information, an initial three-dimensional point cloud map is constructed by a feature matching and pose estimation algorithm based on depth constraints.

[0036] Among them, the depth-constrained feature matching refers to verifying the accuracy of matching by using depth values when performing image feature point matching, and excluding inconsistent mismatched point pairs.

[0037] Among them, the pose estimation algorithm refers to calculating the spatial position and attitude transformation between the current camera and the reference frame, commonly using PnP or ICP methods.

[0038] Among them, the initial three-dimensional point cloud map is a dense point set containing the three-dimensional positions of objects in the environment, which is the preliminary result of mapping.

[0039] In a possible implementation, S3 specifically includes:

[0040] S301: Extract point features (such as ORB) and line features from RGB color images.

[0041] Point features (such as ORB) represent locally salient and recognizable pixels in an image. ORB is an efficient binary descriptor suitable for real-time feature matching.

[0042] Among them, line features are information about straight line segments formed by edges or structures in an image, which can assist in structure recognition and geometric constraints.

[0043] S302: Perform preliminary matching on point features (such as ORB) and line features to obtain preliminary matching feature pairs.

[0044] Among them, the preliminary matching feature pair refers to two feature points from different image frames, which are considered to be projections of the same spatial point.

[0045] S303: Using depth information provided by the RGBD camera, the initial matching feature pairs are verified and filtered for depth consistency to obtain robust matching features.

[0046]

[0047] Where Z1 represents the depth value of the first matching point in its respective camera coordinate system, and Z2 represents the depth value of the second matching point in its respective camera coordinate system. Indicates the depth difference threshold. Represents the absolute value symbol.

[0048] S304: Based on robust matching features, the camera pose transformation matrix of the current frame relative to the previous frame is estimated by solving the PnP problem.

[0049]

[0050] in, Let argmin denote the camera pose transformation matrix of the current frame k relative to the previous frame k-1, and p() denote the robust kernel function. This represents the three-dimensional coordinates of the i-th feature point in the previous frame (k-1). Let T represent the camera's projection function, and let T represent the pose transformation matrix. Indicates the relationship between the current frame k and the... Matched 2D image pixel coordinates This represents the square of the Euclidean distance.

[0051] Among them, the PnP problem refers to the mathematical problem of solving the camera pose given a 3D point and its corresponding image projection point (Perspective-n-Point).

[0052] S305: Based on the camera pose transformation matrix, the initial 3D point cloud map is constructed by transforming the depth map or 3D points acquired in the current frame to the global coordinate system.

[0053] It should be noted that this method ensures the spatial rationality and robustness of feature matching by extracting and matching points and lines, combined with depth consistency filtering. The PnP algorithm uses the filtered robust features to estimate the camera pose transformation, effectively reducing error propagation and supporting accurate registration between different frames. In particular, the introduction of a depth difference threshold constraint improves matching accuracy and avoids false matches caused by occlusion, texture duplication, etc. Finally, the pose matrix unifies the 3D information of the current frame into the global coordinate system, gradually constructing a coherent and accurate 3D point cloud map. The overall process balances geometric accuracy and computational efficiency, providing a solid foundation for subsequent mapping optimization and semantic mapping.

[0054] S4: Using a pre-trained semantic segmentation neural network model, pixel-level segmentation is performed on RGB color images to obtain multiple categories of elderly care equipment.

[0055] Among them, pre-trained semantic segmentation neural network models refer to deep learning models that are pre-trained on large-scale datasets and have strong image understanding capabilities. Commonly used models include DeepLab, SegFormer, and PSPNet.

[0056] It should be noted that by applying a pre-trained semantic segmentation model to RGB images, pixel-level identification and classification of key facilities in elderly care communities can be achieved. Compared to traditional object detection methods, this strategy offers higher accuracy, finely depicting the contours and boundaries of equipment, which is beneficial for subsequent precise alignment with 3D point clouds. Using a pre-trained model also significantly reduces reliance on large amounts of labeled data, accelerates deployment, and enhances the system's adaptability under different lighting and viewing angles. By accurately identifying facilities such as beds, handrails, and stairs, the system not only improves the semantic expressiveness of the map but also lays a high-quality semantic foundation for intelligent applications such as security monitoring, behavior recognition, and service decision-making.

[0057] In one possible implementation, elderly care facilities specifically include: beds, handrails, corridors, stairs, and tables and chairs.

[0058] S5: Combine the various categories of elderly care equipment with their corresponding tags to associate the initial 3D point cloud map with the location of the elderly care equipment.

[0059] In one possible implementation, S5 specifically includes:

[0060] S501: Extract features from the RGB color image and the depth image respectively to obtain RGB features and depth features.

[0061] S502: By introducing a cross-layer cue adapter in the encoder, RGB features and depth features are fused to generate fused features:

[0062]

[0063] Among them, F fused Indicates fusion characteristics, Indicates attention weights. Represents RGB characteristics, Representing depth features, represents the Sigmoid activation function, and MLP represents the multilayer perceptron.

[0064] Among them, the cross-layer cue adapter is a mechanism in neural networks used to fuse information from different modalities or different levels to improve feature representation capabilities.

[0065] Among them, fusion features refer to a unified feature representation generated by combining RGB and depth information, which has both semantic and geometric information.

[0066] S503: The fused features are decoded by the decoder to obtain the semantic mask.

[0067] The semantic mask is the semantic category prediction result for each pixel in the image, usually represented as a probability vector.

[0068] S504: Projects 3D points onto the image plane based on camera pose.

[0069] S505: Based on the projection results, obtain the semantic probability vector corresponding to the pixel in the semantic mask.

[0070] The semantic probability vector is used to describe the probability distribution of a pixel belonging to each semantic category.

[0071] S506: Based on the semantic probability vector, establish a semantic probability distribution vector for each three-dimensional point.

[0072] S507: Combining multi-frame observation results, the semantic probability distribution vector is updated using either the Bayesian update method or the log-space weighted average method.

[0073]

[0074] in, Represents the updated 3D points The semantic probability distribution vector, This indicates the position of a 3D point in the world coordinate system. Indicates the forgetting factor, Representing a three-dimensional point The semantic probability distribution vector, This indicates the number of pixels in the current frame t. The semantic segmentation probability vector.

[0075] Among them, the Bayesian update method or the log-weighted average method is used to fuse information from multiple frames to enhance the robustness of semantic judgment.

[0076] The forgetting factor controls the weighting between historical and new observations, and its typical value range is [value range missing]. .

[0077] S508: Assign final semantic labels to each 3D point based on the updated semantic probability distribution vector.

[0078] The final semantic label refers to the transformation of the probability distribution after multi-frame fusion into a clear category for semantic annotation.

[0079] S509: By binding the final semantic labels of each 3D point to the initial 3D point cloud map, the location of the initial 3D point cloud map and the elderly care equipment can be associated.

[0080] It should be noted that a robust and high-precision 3D semantic annotation mechanism is constructed through multimodal feature fusion and cross-frame semantic updates. The introduction of a cross-layer cue adapter enhances the expressive power of RGB and depth information fusion, enabling the generated fused features to consider both texture and spatial awareness. The connection between 3D points and image pixels is established using projection and semantic masks, and multi-frame observations are integrated through probability distribution modeling and Bayesian updates, significantly reducing the impact of single-frame misjudgments on mapping quality. Finally, each 3D point is assigned a stable and reliable semantic label, achieving semantic enhancement of the point cloud map. This method is highly adaptable to occlusion and viewpoint changes, making it particularly suitable for complex, variable, and dynamic application scenarios in elderly care communities.

[0081] S6: Based on the association results and combined with structured geometric constraints, the initial 3D point cloud map is optimized in the backend using a factor graph model to obtain an optimized 3D point cloud map.

[0082] Among them, the association result refers to the spatial binding relationship between the 3D point cloud completed in the previous stage and the semantic tags of elderly care equipment.

[0083] Among them, structured geometric constraints refer to geometric rules defined based on the characteristics of the elderly care community scenario, such as walls should be flat, handrails should be of uniform height, and furniture should be parallel to the ground, which are used to limit the degrees of freedom for map optimization.

[0084] Among them, the factor graph model is a graph optimization structure that models camera pose and map points as nodes of a graph, and geometric constraints and observation information as edges (factors). It is often used for back-end optimization in SLAM systems.

[0085] Among them, optimized 3D point cloud map refers to the final 3D map after accuracy improvement and error correction, based on the preservation of semantic and geometric information.

[0086] In one possible implementation, the structured geometric constraints include: straight wall constraints in the corridor area, planar parallelism constraints between furniture and the floor in the room, and height consistency constraints for specific facilities (such as handrails).

[0087] In one possible implementation, S6 specifically includes:

[0088] S601: Geometric clustering of point clouds in the initial 3D point cloud map is performed using the RANSAC algorithm or region growing segmentation algorithm to extract typical planar structures and dominant directions.

[0089] Typical planar structures include: floor, walls, and tabletop.

[0090] S602: Based on the typical planar structure and dominant direction, the initial 3D point cloud map backend optimization problem is modeled as a factor graph optimization problem.

[0091] The vertices of the factor graph include all camera poses and map points, while the edges are constraints.

[0092] S603: Determine the overall objective function for the factor graph optimization problem:

[0093]

[0094]

[0095]

[0096]

[0097] in, Describe the overall objective function. The summation symbol, e repro e represents the reprojection error term. plane e represents the point-to-surface distance error term. line Represents the structured line constraint term, e manhattan This indicates the Manhattan World constraint. T represents the square of the Euclidean distance. i P represents the pose of the i-th camera frame. j Let p represent the j-th 3D map point. ij n represents the two-dimensional pixel position of the feature point in the j-th 3D map point actually observed by the i-th camera. n This represents the normal vector of plane n. T Indicates transpose, d nM represents the distance offset of plane n from the origin. k This represents the k-th principal axis in the Manhattan world framework. Indicates the modulus length.

[0098] The main axes include the X, Y, and Z axes.

[0099] S604: With the goal of minimizing the overall objective function, and under the constraints of structured geometric constraints, the factor graph optimization problem is solved through an optimization algorithm to obtain an optimized 3D point cloud map.

[0100] It should be noted that by introducing geometric clustering and factor graph optimization, the structural consistency and spatial accuracy of the 3D point cloud map are greatly improved. By extracting typical planes and dominant directions in the scene, geometric priors are established for subsequent optimization. The mapping problem is modeled as a factor graph with multiple constraints. By defining objective function terms such as reprojection error, point-to-surface distance, structural lines, and Manhattan direction consistency, comprehensive modeling and control of map errors are achieved. Finally, by solving this optimization problem, the generated point cloud map achieves a high degree of consistency in visual observation and geometric structure, which is particularly suitable for environments with regular structures and high scene repetition, such as elderly care communities, providing more accurate and stable spatial support for subsequent navigation, safety monitoring, and auxiliary services.

[0101] In one possible implementation, the optimization algorithm is specifically the Levenberg-Marquardt algorithm.

[0102] S7: Store the optimized 3D point cloud map and the semantic information of the elderly care equipment associated with the optimized 3D point cloud map into a structured semantic map database.

[0103] Among them, the semantic information of elderly care equipment refers to the identified and labeled categories of elderly care facilities (such as beds, armrests, tables and chairs) and their attributes such as location, type, and labels in three-dimensional space.

[0104] Among them, a structured semantic map database is a database that contains multi-dimensional information such as three-dimensional coordinates, semantic labels, and category attributes. It is used to systematically store and quickly query map data, and typically supports graph databases, relational databases, or NoSQL databases.

[0105] It should be noted that by uniformly storing the optimized 3D point cloud and the semantic information of the elderly care equipment in the structured semantic map database, the conversion from data to knowledge is achieved. Compared with traditional maps that only contain geometric coordinates, this database not only retains the accuracy of the spatial structure, but also incorporates rich semantic tags, with stronger queryability and service support capabilities. The structured storage method supports multi-dimensional retrieval by category, region, equipment, etc., which helps to realize multi-scenario applications such as intelligent navigation, auxiliary decision-making, and equipment management. In addition, the semantic map can also serve as the basis for the digital twin of the operation status of the elderly care community, enabling linkage with other information systems (such as security, medical care, and dispatching), and greatly improving the digital management level and service response efficiency of the community.

[0106] S8: In the case of changes in the environment of the elderly care community, through the revisit detection mechanism, the structured semantic map database is incrementally updated.

[0107] Among them, the revisit detection mechanism, also known as loop closure detection, is used to identify whether the current perspective has appeared in the historical data, so as to judge whether it is in a known scene location, and is commonly used for map calibration and update.

[0108] It should be noted that by introducing the revisit detection mechanism, the rapid identification of changes in the elderly care community scene and the incremental update of the semantic map are achieved, significantly improving the environmental adaptability and real-time performance of the system. This mechanism can automatically detect similar frames when the equipment reappears or the environment is partially adjusted, judge whether it enters a known area, and thus trigger the correction of the local map, semantic update, or elimination of redundant data. Compared with the traditional full-scale reconstruction method, the incremental update not only saves computing resources, but also maintains the continuity of the semantic map and the integrity of historical data, which is suitable for scenarios such as elderly care communities where dynamic changes are frequent and the requirement for service continuity is high. This method ensures that the map is always in the latest state, providing a highly reliable data basis for applications such as intelligent navigation, risk warning, and dynamic resource scheduling.

[0109] In the embodiment of the present invention, the RGB-D camera is used to collect the environmental color image and the depth map, and the initial 3D map is quickly generated by feature point matching with depth constraints. Semantic segmentation is used to label objects such as beds, handrails, corridors, and stairs, realizing the semanticization of the map. Further introducing specific geometric constraints in the elderly care community (such as corridor straight line constraints and furniture plane alignment constraints) to optimize the map accuracy, a maintainable semantic map database is generated, which can quickly construct a high-precision and semantically rich community map, providing basic data support for fall detection, intelligent navigation, and community management, and significantly improving the efficiency and reliability of intelligent applications in the elderly care community.

[0110] Refer to the attached Figure 2 , which shows the structural schematic diagram of an RGB-D based intelligent mapping system for elderly care communities provided by the embodiment of the present invention.

[0111] This invention provides an RGB-D-based intelligent mapping system 20 for elderly care communities, comprising: a processor 201 and a memory 202;

[0112] The memory 202 stores programs or instructions that can run on the processor 201. When the program or instructions are executed by the processor 201, they implement the steps of the above-described intelligent mapping method for elderly care communities based on RGB-D and achieve the same technical effect. To avoid repetition, the present invention will not elaborate further.

[0113] It should be understood that the processor 201 in this embodiment of the invention may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0114] It should also be understood that the memory 202 in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DR RAM).

[0115] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0116] It should be understood that, in various embodiments of the present invention, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0117] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0118] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0119] In the embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0120] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0121] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0122] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0123] This invention provides a readable storage medium comprising: storing a program or instructions on the readable storage medium, wherein when the program or instructions are executed by a processor, the program or instructions implement the steps of the above-described intelligent mapping method for elderly care communities based on RGB-D, and can achieve the same technical effect. To avoid repetition, this invention will not elaborate further.

[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the protection scope of the present invention.

Claims

1. A method for intelligent mapping of elderly care communities based on RGB-D, characterized in that, include: S1: Collect environmental images of the elderly care community, wherein the environmental images include RGB color images and depth images; S2: Process the environmental image to generate visual feature points and depth information; S3: Based on the visual feature points and the depth information, construct an initial 3D point cloud map using a feature matching and pose estimation algorithm based on depth constraints; S4: The RGB color image is segmented at the pixel level using a pre-trained semantic segmentation neural network model to obtain multiple categories of elderly care equipment; S5: Combine the various categories of elderly care equipment and the corresponding tags of the elderly care equipment to associate the initial three-dimensional point cloud map with the location of the elderly care equipment; S6: Based on the association results and combined with structured geometric constraints, the initial 3D point cloud map is optimized in the backend using a factor graph model to obtain an optimized 3D point cloud map. S7: Store the optimized 3D point cloud map and the semantic information of the elderly care equipment associated with the optimized 3D point cloud map into a structured semantic map database; S8: When the environment of the elderly care community changes, the structured semantic map database is incrementally updated through a revisit detection mechanism.

2. The intelligent mapping method for elderly care communities based on RGB-D according to claim 1, characterized in that, Following S1, the following is also included: The depth image is preprocessed by denoising and filtering.

3. The intelligent mapping method for elderly care communities based on RGB-D according to claim 1, characterized in that, S3 specifically includes: S301: Extract point features (such as ORB) and line features from the RGB color image; S302: Perform preliminary matching on the point features (such as ORB) and the line features to obtain preliminary matching feature pairs; S303: Using the depth information provided by the RGBD camera, perform depth consistency verification and screening on the preliminary matching feature pairs to obtain robust matching features: ； Where Z1 represents the depth value of the first matching point in its respective camera coordinate system, and Z2 represents the depth value of the second matching point in its respective camera coordinate system. Indicates the depth difference threshold. Indicates the absolute value symbol; S304: Based on the robust matching features, the camera pose transformation matrix of the current frame relative to the previous frame is estimated by solving the PnP problem: ； in, Let argmin denote the camera pose transformation matrix of the current frame k relative to the previous frame k-1, and p() denote the robust kernel function. This represents the three-dimensional coordinates of the i-th feature point in the previous frame (k-1). Let T represent the camera's projection function, and let T represent the pose transformation matrix. Indicates the relationship between the current frame k and the... Matched 2D image pixel coordinates Represents the square of the Euclidean distance; S305: Based on the camera pose transformation matrix, the initial 3D point cloud map is constructed by transforming the depth map or 3D points acquired in the current frame to the global coordinate system.

4. The intelligent mapping method for elderly care communities based on RGB-D according to claim 1, characterized in that, The elderly care equipment specifically includes: beds, handrails, corridors, stairs, and tables and chairs.

5. The intelligent mapping method for elderly care communities based on RGB-D according to claim 1, characterized in that, S5 specifically includes: S501: Perform feature extraction on the RGB color image and the depth image respectively to obtain RGB features and depth features; S502: By introducing a cross-layer cue adapter in the encoder, the RGB features and the depth features are fused to generate a fused feature: ； Among them, F fused Indicates fusion features, Indicates attention weights. Represents RGB characteristics, Representing depth features, represents the Sigmoid activation function, and MLP represents the multilayer perceptron. S503: The fused features are decoded using a decoder to obtain a semantic mask; S504: Based on camera pose, project 3D points onto the image plane; S505: Based on the projection results, obtain the semantic probability vector corresponding to the pixel in the semantic mask; S506: Based on the semantic probability vector, establish a semantic probability distribution vector for each of the three-dimensional points; S507: Combining multi-frame observation results, the semantic probability distribution vector is updated using either the Bayesian update method or the log-space weighted average method. ； in, Represents the updated 3D points The semantic probability distribution vector, This indicates the position of a 3D point in the world coordinate system. Indicates the forgetting factor, Representing a three-dimensional point The semantic probability distribution vector, This indicates the number of pixels in the current frame t. semantic segmentation probability vector; S508: Assign final semantic labels to each of the three-dimensional points based on the updated semantic probability distribution vector; S509: By binding the final semantic labels of each three-dimensional point to the initial three-dimensional point cloud map, the location of the initial three-dimensional point cloud map and the elderly care equipment can be associated.

6. The intelligent mapping method for elderly care communities based on RGB-D according to claim 1, characterized in that, The structured geometric constraints include: straight wall constraints in the corridor area, planar parallelism constraints between furniture and the floor in the room, and height consistency constraints for specific facilities (such as handrails).

7. The intelligent mapping method for elderly care communities based on RGB-D according to claim 1, characterized in that, S6 specifically includes: S601: Perform geometric clustering on the point cloud in the initial 3D point cloud map using the RANSAC algorithm or the region growing segmentation algorithm to extract typical planar structures and dominant directions; S602: Based on the typical planar structure and the dominant direction, the initial 3D point cloud map backend optimization problem is modeled as a factor graph optimization problem; S603: Determine the overall objective function of the factor graph optimization problem: ； ； ； ； in, Describe the overall objective function. The summation symbol, e repro e represents the reprojection error term. plane e represents the point-to-surface distance error term. line Represents the structured line constraint term, e manhattan This indicates the Manhattan World constraint. T represents the square of the Euclidean distance. i P represents the pose of the i-th camera frame. j Let p represent the j-th 3D map point. ij n represents the two-dimensional pixel position of the feature point in the j-th 3D map point actually observed by the i-th camera. n Describes the normal vector of plane n. T Indicates transpose, d n M represents the distance offset of plane n from the origin. k This represents the k-th principal axis in the Manhattan world framework. Indicates the modulus length; S604: With the goal of minimizing the overall objective function, and under the constraints of the structured geometric constraints, the factor graph optimization problem is solved by an optimization algorithm to obtain the optimized 3D point cloud map.

8. The intelligent mapping method for elderly care communities based on RGB-D according to claim 7, characterized in that, The optimization algorithm specifically refers to the Levenberg-Marquardt algorithm, etc.

9. A smart mapping system for elderly care communities based on RGB-D, characterized in that, include: Processor and memory; The memory stores programs or instructions that can run on the processor, and when the program or instructions are executed by the processor, they implement the steps of the RGB-D-based intelligent mapping method for elderly care communities as described in any one of claims 1 to 8.

10. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the RGB-D-based intelligent mapping method for elderly care communities as described in any one of claims 1 to 8.

Images