Infant caregiver upper limb region image segmentation and restoration method, device and equipment
Through frame processing and precise segmentation technology in video, identifying and repairing the upper limb areas of infant caregivers, the realism and continuity problems in dynamic videos are solved, and the naturalness and fluency of the video are improved.
Patent Information
- Application Number
- CN202411421509.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-12
- Publication Date
- 2024-11-12
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The prior art is difficult to maintain the overall reality and continuity of the video when processing dynamic videos, especially when it is necessary to automatically repair the caregiver's upper limb area in infant and young children's care scenarios, visual incongruity and content discontinuity often occur.
By acquiring video data and decomposing it into multi-frame images, the image segmentation model is used to determine the location information of the upper limbs of infant caregivers, combining the depth analysis model to distinguish the foreground and background areas, and repairing it using pixel filling technology to ensure the naturalness and continuity of the image.
It realizes accurate identification and repair of the upper limb areas of infant caregivers in dynamic videos, reduces interference from other elements in the video, avoids image distortion or jumping, and improves the visual quality and user experience of the video.
Smart Images

Figure CN118941584A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image restoration, and in particular to a method, device and equipment for segmenting and restoring images of the upper limb region of an infant caregiver. Background Art
[0002] In the field of video processing and image editing, how to efficiently and naturally repair segmented images has always been a technical challenge. Especially when complex dynamic backgrounds are involved, the repair effect is often prone to visual disharmony. Traditional image repair methods usually rely on global models or pre-set texture filling strategies. Although they perform well in static images, they are prone to problems such as content discontinuity and color mismatch when processing dynamic videos.
[0003] In video editing of infant care scenes, it is usually necessary to remove or hide the exposed limbs of the infant caregiver when interacting with the infant to create creative visual effects, such as making the infant appear to be flying in the air. However, existing technical means often require a lot of manual intervention and complex calculations to achieve this effect, which is time-consuming and difficult to ensure the naturalness and coherence of the final effect. Therefore, there is an urgent need for a method that can automatically and naturally perform image restoration on the segmented arm area of the infant caregiver to improve the efficiency and quality of video editing and provide better technical support for creative content generation.
[0004] Therefore, how to maintain the overall realism and continuity of the video when processing dynamic video data for image restoration is an urgent problem to be solved. Summary of the invention
[0005] In view of this, the present invention provides a method, device and equipment for image segmentation and restoration of the upper limb area of an infant caregiver, so as to solve the problem in the prior art that it is impossible to maintain the overall realism and continuity of the video when processing dynamic video data for image restoration.
[0006] The technical solution adopted by the present invention is: In a first aspect, the present invention provides a method for segmenting and repairing an image of an upper limb region of an infant caregiver, the method comprising: S1: Acquire video data of an infant care scene, and decompose the video data into multiple frames of images, wherein the infant care scene includes an infant and an infant caregiver; S2: performing image segmentation on each of the images to determine the position information of the upper limb region of the infant caregiver and the position information of the upper limb region of the infant; S3: performing depth analysis on pixel points in the upper limb area of the infant caregiver according to the upper limb area position information of the infant caregiver to determine the foreground area position information and the background area position information; S4: Based on the foreground area position information and the background area position information, combined with the infant upper limb area position information, the foreground area and the background area are respectively filled with pixels to complete the image restoration of the upper limb area of the infant caregiver.
[0007] Preferably, the S2 includes: S21: inputting each of the images into a pre-trained target detection model, and outputting initial position information of the upper limbs of the infant caregiver and initial position information of the upper limbs of the infant; S22: Input each of the images into a pre-trained image segmentation model, perform secondary correction on the infant caregiver upper limb detection frame position information and the infant upper limb detection frame position information, and determine the infant caregiver upper limb region position information and the infant upper limb region position information.
[0008] Preferably, S3 includes: S31: inputting each of the images into a pre-trained depth calculation model to output global depth information; S32: Determine depth information of the upper limb region of the infant caregiver based on the position information of the upper limb region of the infant caregiver and the global depth information; S33: extracting contour features of the upper limb region of the infant caregiver based on the position information of the upper limb region of the infant caregiver, and determining the contour position information of the upper limb region of the infant caregiver; S34: determining the contour depth information of the upper limb area of the infant caregiver according to the depth information of the upper limb area of the infant caregiver and the contour position information of the upper limb area of the infant caregiver; S35: According to preset rules, depth analysis is performed on the contour depth information of the upper limb area of the infant caregiver to determine the background area position information and the foreground area position information.
[0009] Preferably, the S35 includes: S351: selecting a pixel point on the contour of the upper limb area of the infant caregiver as the central pixel point, and taking the depth value corresponding to the central pixel point as the reference depth value according to the depth information of the contour of the upper limb area of the infant caregiver; S352: taking depth values of a plurality of target pixel points within a preset range around the central pixel point as target depth values, performing difference calculation between each target depth value and the reference depth value, and determining each depth difference; S353: Classifying the central pixel point as a background point or a foreground point according to each of the depth differences and a preset depth threshold; S354: Repeat steps S351 to S353 for all pixel points on the contour of the upper limb area of the infant caregiver until all pixel points are classified into background points and foreground points; S355: Determine the background area position information and the foreground area position information according to the background point position information and the foreground point position information.
[0010] Preferably, the S353 includes: S3531: Compare each depth difference with a preset depth threshold, and determine the proportion of target pixel points whose depth difference is greater than the depth threshold; S3532: If the target pixel ratio is greater than a preset ratio threshold, classify the central pixel as a background point; S3533: If the target pixel ratio is less than or equal to a preset ratio threshold, the central pixel is classified as a foreground point.
[0011] Preferably, the S4 includes: S41: judging whether a second pixel point in the upper limb region of the infant is located around a first pixel point in the upper limb region of the infant caregiver according to the upper limb region position information of the infant caregiver and the upper limb region position information of the infant; S42: If the first pixel point is surrounded by the second pixel point, the first pixel point is removed and no pixel filling is performed; S43: If there are no second pixels around the first pixel, perform pixel filling on the first pixel.
[0012] Preferably, the S43 includes: S431: If there is no second pixel around the first pixel, determine that the first pixel is a background point or a foreground point, and obtain a preset domain pixel window; S432: If the first pixel is a background point, the color channel average of all pixels in the domain pixel window is used as the color channel value of the first pixel; S433: If the first pixel point is a foreground point, a preset approximate nearest neighbor search algorithm is used to determine a best matching area in the domain pixel window, and the texture information of the best matching area is filled into the first pixel point.
[0013] In a second aspect, the present invention provides a device for segmenting and repairing an image of an upper limb region of an infant caregiver, the device comprising: An image decomposition module, used to obtain video data in an infant care scene, and decompose the video data into multiple frames of images, wherein the infant care scene includes an infant and an infant caregiver; An image segmentation module, used to perform image segmentation on each of the images to determine the position information of the upper limb area of the infant caregiver and the position information of the upper limb area of the infant; A depth analysis module, configured to perform depth analysis on pixel points in the upper limb region of the infant caregiver based on the upper limb region position information of the infant caregiver, and determine foreground region position information and background region position information; The image restoration module is used to fill pixels in the foreground area and the background area respectively according to the foreground area position information and the background area position information, combined with the infant upper limb area position information, to complete the image restoration of the upper limb area of the infant caregiver.
[0014] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising: at least one processor, at least one memory, and computer program instructions stored in the memory, which, when executed by the processor, implement the method of the first aspect in the above-mentioned embodiment.
[0015] In a fourth aspect, an embodiment of the present invention further provides a storage medium on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the method of the first aspect in the above-mentioned embodiment is implemented.
[0016] In summary, the beneficial effects of the present invention are as follows: The present invention provides a method, device, equipment and medium for segmenting and repairing images of the upper limb region of an infant caregiver. The method comprises: acquiring video data in an infant care scene, and decomposing the video data into multiple frames of images; performing image segmentation on each of the images to determine position information of the upper limb region of the infant caregiver and position information of the upper limb region of the infant; performing a depth analysis on pixel points in the upper limb region of the infant caregiver based on the position information of the upper limb region of the infant caregiver to determine position information of the foreground region and position information of the background region; performing pixel filling on the foreground region and the background region respectively based on the position information of the foreground region and the background region in combination with the position information of the upper limb region of the infant to complete image repair of the upper limb region of the infant caregiver. The present invention decomposes video data into multiple frames of images, performs independent image segmentation and analysis on each frame, identifies the upper limb area of the infant caregiver and the infant, and performs depth analysis on the pixel points of the upper limb area of the infant caregiver to accurately distinguish the foreground and background areas. This precise area positioning and depth analysis can dynamically adapt to changes in the scene when processing dynamic videos, reducing interference with other elements in the video. At the same time, based on the position information of the foreground and background, combined with the position information of the infant's upper limbs, the foreground and background areas are pixel-filled respectively, while ensuring image restoration, the continuity and naturalness of the video are maintained, and the common image distortion or jump phenomenon in traditional methods is avoided. Through meticulous partition processing and frame-by-frame restoration, the details and smoothness of the video can be retained to the maximum extent, making the restored video look more natural and real, thereby improving the visual quality of the video and user experience. In short, when processing dynamic video data, the scheme not only improves the accuracy of image restoration, but also maintains the overall realism and continuity of the video through frame processing, precise segmentation and intelligent filling. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] In order to more clearly illustrate the technical solution of the embodiment of the present invention, the drawings required for use in the embodiment of the present invention will be briefly introduced below. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work, and these are all within the protection scope of the present invention.
[0018] Figure 1 It is a schematic diagram of the overall working process of the method for segmenting and repairing the upper limb area image of an infant caregiver in Example 1 of the present invention; Figure 2 It is a schematic diagram of a process for determining the upper limb region position information of the infant caregiver and the upper limb region position information of the infant in Embodiment 1 of the present invention; Figure 3 This is a schematic diagram of a process of performing depth analysis on pixel points in the upper limb area of an infant caregiver in Embodiment 1 of the present invention; Figure 4 This is a schematic diagram of a process for performing depth analysis on the contour depth information of the upper limb area of the infant caregiver in Embodiment 1 of the present invention; Figure 5 It is a schematic diagram of a process of determining the background area position information and the foreground area position information according to the background point position information and the foreground point position information in Embodiment 1 of the present invention; Figure 6 This is a schematic diagram of a process of respectively performing pixel filling on a foreground area and a background area in Embodiment 1 of the present invention; Figure 7 This is a schematic diagram of a process of performing pixel filling on a first pixel point in Embodiment 1 of the present invention; Figure 8 It is a structural block diagram of the upper limb region image segmentation and restoration device for infant caregivers in Embodiment 2 of the present invention; Fig. 9 It is a schematic diagram of the structure of the electronic device in Embodiment 3 of the present invention. DETAILED DESCRIPTION
[0019] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution in the embodiment of the present invention will be clearly and completely described in conjunction with the drawings in the embodiment of the present invention. It should be noted that, in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. In the description of the present invention, it should be understood that the orientation or position relationship indicated by the terms "center", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc. is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present invention. Moreover, the term "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such a process, method, article or device. In the absence of further restrictions, the elements defined by the phrase "comprising..." do not exclude the existence of other identical elements in the process, method, article or device comprising the elements. If there is no conflict, the embodiments of the present invention and the various features in the embodiments can be combined with each other, all within the protection scope of the present invention.
[0020] Example 1
[0021] See also Figure 1 Embodiment 1 of the present invention discloses a method for segmenting and repairing an image of an upper limb region of an infant caregiver, the method comprising: S1: Acquire video data of an infant care scene, and decompose the video data into multiple frames of images, wherein the infant care scene includes an infant and an infant caregiver; Specifically, in the infant care scenario, specially shot video data is obtained and decomposed into multiple continuous frame images. The video data comes from a surveillance camera or a mobile device and is used to capture various scenes occurring in the infant care process. By decomposing a continuous video stream into multiple frame images, every moment in the video can be converted into a static image, thereby providing more refined basic data for subsequent image processing tasks, such as image segmentation, deep analysis, and repair operations. The infant caregivers include parents, professional childcare workers, nursery staff, and any adults who provide care and education; they may also include robots with the function of monitoring infants. Frame-by-frame processing ensures that the movement details of infant caregivers and infants can be accurately captured and processed in a dynamic environment, which helps to improve the accuracy and effectiveness of monitoring and analysis in practical applications.
[0022] S2: performing image segmentation on each of the images to determine the position information of the upper limb region of the infant caregiver and the position information of the upper limb region of the infant; Specifically, in the process of processing multiple frames of images, image segmentation technology is applied to identify and distinguish different areas in the image, especially the upper limb area of the infant caregiver and the upper limb area of the infant. The image segmentation technology includes a technology for decomposing an image into multiple meaningful areas or objects, which is used here to accurately locate the upper limb positions of the infant caregiver and the infant. By analyzing the pixel data in each frame of the image, the upper limb parts belonging to the infant caregiver and the infant are accurately identified, and the position information of these areas is generated. The position information includes the boundaries, shapes and positions of the areas, etc., which serve as the key input for subsequent deep analysis and image restoration, ensuring that even in complex care scenes, the actions of the infant caregiver and the infant can be distinguished and accurately located, providing accurate and reliable basic data for subsequent processing steps.
[0023] In one embodiment, see Figure 2 , said S2 comprises: S21: inputting each of the images into a pre-trained target detection model, and outputting initial position information of the upper limbs of the infant caregiver and initial position information of the upper limbs of the infant; Specifically, each of the images is input into a pre-trained target detection model, such as the YoloV8s model, which has been trained with a large amount of data and can accurately identify and locate specific targets in the image, namely, the upper limb area of the infant caregiver and the upper limb area of the infant. The target detection model outputs the initial position information of the upper limb of the infant caregiver and the initial position information of the upper limb of the infant by analyzing the features in the image, such as edges, textures, and colors. The initial position information is represented in the form of a detection frame, which frames the area in the image related to the upper limb of the infant caregiver and the upper limb of the infant. This process provides preliminary identification and positioning of the target area, laying the foundation for subsequent more accurate image segmentation and position correction.
[0024] S22: Input each of the images into a pre-trained image segmentation model, perform secondary correction on the infant caregiver upper limb detection frame position information and the infant upper limb detection frame position information, and determine the infant caregiver upper limb region position information and the infant upper limb region position information.
[0025] Specifically, after obtaining the preliminary upper limb position information, these detection frames are further input into a pre-trained image segmentation model, such as SAM, which can perform more detailed segmentation of the area within the detection frame and accurately extract the upper limb area of the infant caregiver and infant. Through the secondary correction process, the initial detection frame position information can be further optimized to reduce the regional deviation caused by the target detection model error, ensuring that the segmentation result is more accurate. Finally, the infant caregiver upper limb area position information and infant upper limb area position information output by the segmentation model not only include the boundaries of the area, but also accurately determine the shape and details of these areas, providing more accurate basic data for subsequent in-depth analysis and image restoration operations.
[0026] S3: performing depth analysis on pixel points in the upper limb area of the infant caregiver according to the upper limb area position information of the infant caregiver to determine the foreground area position information and the background area position information; Specifically, using the previously obtained position information of the upper limb area of the infant caregiver, the pixel points in the upper limb area of the infant caregiver are subjected to depth analysis to further distinguish the foreground and background areas. The depth analysis includes analyzing the depth information of each pixel point in the image, that is, the distance relationship from the camera, to determine which pixel areas belong to the foreground area and which pixel areas belong to the background area. The analysis process involves using a depth image obtained by a depth sensor, or calculating the depth information through multi-view images. During the analysis process, the foreground and background areas are identified and distinguished based on features such as depth value, color, and texture. This fine separation step ensures that the true shape and details of the upper limbs of the infant caregiver can be accurately preserved in the subsequent image restoration and pixel filling process, avoiding unnecessary interference with the background area, thereby maintaining the overall naturalness and visual continuity of the video.
[0027] In one embodiment, see Figure 3 , said S3 comprises: S31: inputting each of the images into a pre-trained depth calculation model to output global depth information; Specifically, each frame of the image is input into a pre-trained depth calculation model, such as MobileDepth, which is used to calculate the global depth information of the image. The depth calculation model uses the depth estimation algorithm learned during its training process to generate the depth value of each pixel, thereby constructing the depth field of the entire image. The global depth information includes the distance of each point in the image relative to the camera. This depth information can provide a comprehensive spatial understanding of the scene and provide basic data for subsequent local depth analysis and detail processing.
[0028] S32: Determine depth information of the upper limb region of the infant caregiver based on the position information of the upper limb region of the infant caregiver and the global depth information; Specifically, after obtaining the global depth information, the depth information in the area is extracted in combination with the position information of the upper limb area of the infant or child caregiver. By pairing the position information of the upper limb of the infant or child caregiver with the global depth field, the depth value of each pixel in the upper limb area of the infant or child caregiver can be accurately extracted. By filtering and analyzing the depth data of only the upper limb area of the infant or child caregiver in the image, the specific depth information of the area is obtained, including the spatial position of the infant or child caregiver's arms and hands relative to the background.
[0029] S33: extracting contour features of the upper limb region of the infant caregiver based on the position information of the upper limb region of the infant caregiver, and determining the contour position information of the upper limb region of the infant caregiver; Specifically, after obtaining the depth information of the upper limb area of the infant or child caregiver, it is necessary to further extract the contour features of the infant or child caregiver's upper limbs. By analyzing the depth data in the upper limb area of the infant or child caregiver, the boundaries and contours of the hands and arms are identified. Contour feature extraction includes edge detection or contour detection of the depth field to identify the shape contours of the infant or child caregiver's upper limbs. These contour position information are crucial to the subsequent image restoration process. They help define and accurately locate the shape and boundaries of the upper limbs, ensuring that the image restoration can reconstruct the actual appearance of the infant or child caregiver's upper limbs.
[0030] S34: determining the contour depth information of the upper limb area of the infant caregiver according to the depth information of the upper limb area of the infant caregiver and the contour position information of the upper limb area of the infant caregiver; Specifically, the depth information and contour position information of the upper limb area of the infant caregiver are combined to determine the depth information of the contour of the upper limb area of the infant caregiver. First, the depth values related to the hands and arms of the infant caregiver are accurately screened out from the global depth data of the upper limb area of the infant caregiver extracted in the previous step. These depth values provide the relative distance between each pixel and the camera. Next, the contour position information of the upper limb area of the infant caregiver obtained by contour extraction is used to apply these contour data to the depth field to extract the depth value at the contour. In this way, the depth information of the boundary of the area can be obtained, including the accurate depth of the contour of the hands and arms. This process ensures that the contour depth information not only reflects the spatial position of the hands and arms, but also accurately displays the depth changes of the contour boundaries.
[0031] S35: According to preset rules, depth analysis is performed on the contour depth information of the upper limb area of the infant caregiver to determine the background area position information and the foreground area position information.
[0032] Specifically, first, based on the relative distance between each pixel in the contour depth information and the camera, preset rules such as depth threshold, depth difference or depth gradient are applied to distinguish the foreground and background areas. These rules can identify which areas belong to the foreground area and which areas belong to the background area according to the changes in the depth information. Through depth analysis, the pixels in the upper limb area of the infant caregiver are effectively separated into foreground and background by depth, ensuring that in the subsequent image processing and restoration, the details of the foreground can be accurately maintained and the background area can be reasonably processed, thereby improving the overall quality and naturalness of the image restoration.
[0033] In one embodiment, see Figure 4 , the S35 comprises: S351: selecting a pixel point on the contour of the upper limb area of the infant caregiver as the central pixel point, and taking the depth value corresponding to the central pixel point as the reference depth value according to the depth information of the contour of the upper limb area of the infant caregiver; Specifically, a pixel is selected from the contour of the upper limb area of the infant caregiver as the central pixel. According to the obtained contour depth information, the depth value of the central pixel is recorded, and this value will be used as the reference depth value for subsequent classification. The reference depth value is a benchmark for comparison with the depth values of the surrounding pixels to help determine whether the central point is in the foreground or background area. In this way, a standard can be set for each central pixel to facilitate depth difference calculation and subsequent point classification.
[0034] S352: taking depth values of a plurality of target pixel points within a preset range around the central pixel point as target depth values, performing difference calculation between each target depth value and the reference depth value, and determining each depth difference; Specifically, the depth values of pixels within a preset range around the central pixel are extracted. For example, these surrounding target pixels are located within a W*W (W=5) pixel range centered on the central point. Next, the depth value of each target pixel is subtracted from the reference depth value of the central pixel to obtain the difference between each target depth value and the reference depth value, i.e., each depth difference.
[0035] S353: Classifying the central pixel point as a background point or a foreground point according to each of the depth differences and a preset depth threshold; Specifically, the calculated depth difference is compared with a preset depth threshold, and through the comparison result, the foreground and background attributes of the central pixel are determined based on the depth difference.
[0036] In one embodiment, see Figure 5 , the S353 includes: S3531: Compare each depth difference with a preset depth threshold, and determine the proportion of target pixel points whose depth difference is greater than the depth threshold; Specifically, after calculating the depth difference of each target pixel, these depth differences are compared with a preset depth threshold. The depth threshold is a set reference value used to judge the significance of the depth difference. For example, the depth threshold can be dynamically set to twice the depth value of the center pixel. By counting the number of target pixels with a depth difference greater than the preset threshold within the W*W range around the center pixel, the proportion of these target pixels is calculated. This proportion indicates how many pixels around the center pixel have a depth significantly higher than the reference depth value, providing a preliminary basis for judging whether the center pixel is in a hollow state.
[0037] S3532: If the target pixel ratio is greater than a preset ratio threshold, classify the central pixel as a background point; Specifically, after calculating the ratio of the depth difference of the target pixel points that is greater than the preset depth threshold, it is compared with the preset ratio threshold. For example, the ratio threshold can be set to 1 / 3. If the calculated ratio is greater than the preset ratio threshold, it means that around the central pixel point, the depth values of most target pixels are much greater than the depth of the central pixel point. This indicates that the central pixel point may be in a hollow state, blocking the background pixels in the distance. In this case, the central pixel point is classified as a background point. This classification helps to identify and separate the background area in the image, providing accurate background information for subsequent image processing.
[0038] S3533: If the target pixel ratio is less than or equal to a preset ratio threshold, the central pixel is classified as a foreground point.
[0039] Specifically, on the contrary, if the calculated target pixel ratio is less than or equal to the preset ratio threshold, it means that the depth values of most target pixels around the central pixel are close to the reference depth value and do not reach a significant difference, which indicates that the central pixel may block the foreground pixel with a depth value close to it, such as the part of the infant. In this case, the central pixel is classified as a foreground point. This classification method ensures that the possible foreground area is accurately identified, providing the necessary data support for subsequent image restoration and foreground processing.
[0040] S354: Repeat steps S351 to S353 for all pixel points on the contour of the upper limb area of the infant caregiver until all pixel points are classified into background points and foreground points; Specifically, steps S351 to S353 are applied one by one to all pixel points on the contour of the upper limb area of the infant caregiver, and each pixel point is selected as the central pixel point, and the depth value reference and difference calculation are performed, and classification is performed according to the preset depth threshold, traversing the entire contour area until all pixel points are accurately classified as background points or foreground points. Through this classification method, the attributes of each pixel point in the contour area can be comprehensively determined, ensuring that the final classification result can effectively support subsequent image restoration and processing tasks.
[0041] S355: Determine the background area position information and the foreground area position information according to the background point position information and the foreground point position information.
[0042] Specifically, for all contour points classified as background points, all pixels in the area surrounded by the contour points are marked as background points. This is because the depth information of the background points indicates that they block the distant background. Therefore, the area surrounded by the contour points should naturally belong to the background area. Correspondingly, for contour points classified as foreground points, all pixels in the area surrounded by them are also marked as foreground points, because these foreground points block foreground pixels with similar depth information, such as the upper limbs of the infant caregiver or the hands of the infant. In this step, the background and foreground areas in the image are effectively divided through the overall marking of the area, providing a clear area division basis for subsequent image restoration and processing.
[0043] S4: Based on the foreground area position information and the background area position information, combined with the infant upper limb area position information, the foreground area and the background area are respectively filled with pixels to complete the image restoration of the upper limb area of the infant caregiver.
[0044] Specifically, using the foreground area position information and the background area position information, the background part of the upper limb area of the infant caregiver is filled with pixel values consistent with the background area to restore the natural appearance of its background, which includes filling the background content that was originally obscured in the upper limb area of the infant caregiver. Secondly, for the foreground area, combined with the position information of the infant's upper limb, the foreground part of the upper limb area of the infant caregiver is filled to ensure that the pixels of these areas are consistent with the actual foreground content, such as the details and contours of the infant caregiver's arm. This filling operation can not only repair the missing parts of the upper limb area of the infant caregiver, but also ensure the overall realism and visual coherence of the image, so that the final restoration effect is natural and integrated with the surrounding environment.
[0045] In one embodiment, see Figure 6 , said S4 comprises: S41: judging whether a second pixel point in the upper limb region of the infant is located around a first pixel point in the upper limb region of the infant caregiver according to the upper limb region position information of the infant caregiver and the upper limb region position information of the infant; Specifically, based on the previously obtained upper limb region position information of the infant caregiver and the upper limb region position information of the infant, each first pixel point in the upper limb region of the infant caregiver is checked to determine whether it is surrounded by second pixel points in the upper limb region of the infant. Specifically, this involves checking, for each first pixel point in the upper limb region of the infant caregiver, whether the surrounding pixel points belong to the upper limb region of the infant. This method can identify those pixel points in the upper limb region of the infant caregiver that overlap with the infant's arm, ensuring that the surrounding infant pixel area can be correctly considered when processing these pixel points.
[0046] S42: If the first pixel point is surrounded by the second pixel point, the first pixel point is removed and no pixel filling is performed; Specifically, if it is found in step S41 that the first pixel is surrounded by the second pixel, that is, the area around the first pixel belongs to the upper limb area of the infant, then these first pixels should be removed. The purpose of removing these pixels is to avoid an unnatural "void" effect in the upper limb area of the infant caregiver or mixing with the actual upper limb area of the infant. Since these pixels are actually adjacent to the upper limbs of the infant, excluding them can avoid unnecessary pixel filling in these areas, ensure the accuracy and naturalness of image restoration, and prevent artifacts or visual inconsistencies in the hand area of the infant caregiver.
[0047] S43: If there are no second pixels around the first pixel, perform pixel filling on the first pixel.
[0048] Specifically, each first pixel point is processed. If it is not surrounded by second pixel points in the upper limb area of the infant, the pixel point needs to be filled. These first pixel points can be background points or foreground points, and the specific processing method depends on their classification.
[0049] In one embodiment, see Figure 7 , the S43 comprises: S431: If there is no second pixel around the first pixel, determine that the first pixel is a background point or a foreground point, and obtain a preset domain pixel window; Specifically, check whether there is a second pixel around the first pixel. If there is no second pixel around, determine the category of the first pixel, that is, whether it is a background point or a foreground point. According to its category, determine a preset domain pixel window, usually W*W (W=5) in size. This window will be used for subsequent filling operations to ensure the regional consistency and naturalness of pixel filling.
[0050] S432: If the first pixel is a background point, the color channel average of all pixels in the domain pixel window is used as the color channel value of the first pixel; Specifically, if the first pixel is determined to be a background point, the color channel means of all background points are calculated within the preset domain pixel window. Specifically, the means of the RGB color channels of each pixel in the window are calculated, and these means are used to update the RGB value of the current background point. This method helps to smooth the color transition of the background area and avoid obvious color inconsistencies or abruptness during the filling process.
[0051] S433: If the first pixel point is a foreground point, a preset approximate nearest neighbor search algorithm is used to determine a best matching area in the domain pixel window, and the texture information of the best matching area is filled into the first pixel point.
[0052] Specifically, if the first pixel is determined to be a foreground point, more complex texture filling is required. Use a preset approximate nearest neighbor search algorithm (such as the PatchMatch algorithm) to find the best matching block within the domain pixel window. The algorithm obtains the best matching block by searching for an area with high texture similarity in the foreground area. Then, the texture information of the best matching block is filled into the current foreground point to ensure that the visual effect of the foreground area is natural and coherent. This method can effectively restore the texture details of the foreground area, making the image restoration more realistic and consistent.
[0053] In one embodiment, after S4, the method further includes: S51: performing scoring processing on the restored upper limb region image of the infant or child caregiver according to the preset image evaluation index, and outputting the scoring value of each region; Specifically, in the restored images of the upper limb areas of infants and young children, the images are scored by region based on preset image evaluation indicators. Specifically, the restored images are first divided into several small areas, which are divided according to criteria such as geographical location, feature similarity or structural integrity. Next, each area is scored using an image quality evaluation algorithm, such as SSIM, PSNR or other deep learning model evaluation indicators. These indicators can quantify the image's performance in terms of clarity, color consistency, texture fineness, etc., thereby calculating a score for each area. This score reflects the relative level of the restoration effect of the area and the overall image quality, which is helpful for subsequent decision-making.
[0054] S52: Compare the score value of each area with a preset score threshold, and segment the image corresponding to the area with a score value less than the preset score threshold as a target image; Specifically, each score value is compared with a preset score threshold to identify areas with poor restoration effects. The preset score threshold is set through experience or statistical methods in the previous image evaluation process, and is intended to reflect the minimum acceptance standard of image quality. For areas with a score value less than this threshold, they will be automatically segmented to form target images. These target images often contain parts with unsatisfactory restoration effects. These areas are still blurred, inconsistent in color, or missing textures. Therefore, these target areas are extracted separately to facilitate fine subsequent processing, thereby ensuring the overall quality of the final image.
[0055] S53: Repeat steps S2 to S4 for the target image until the score value of the target image is greater than or equal to the score threshold, and then rejoin the target image to the original restored image of the upper limb area of the infant caregiver.
[0056] Specifically, for the selected target image, the operations of steps S2 to S4 are repeated. First, the target image is re-segmented in detail, the foreground and background are identified, the depth information is analyzed, and the real area to be repaired is carefully determined. Then, with the help of the pixel filling module, the aforementioned area is appropriately filled and textured to improve the repair effect. At the end of each round of iteration, the repaired target area is scored again to determine whether its repair effect reaches the established score threshold. When the score value of the target image finally reaches or exceeds the threshold, these repaired areas are re-joined to the original repaired image. This process not only enhances the overall consistency and aesthetics of the image, but also ensures the adaptability and feedback mechanism of the processing system, so that it can continuously optimize the repair effect in practical applications.
[0057] Example 2
[0058] See also Figure 8 Embodiment 2 of the present invention further provides a device for segmenting and repairing an image of an upper limb region of an infant caregiver, the device comprising: An image decomposition module, used to obtain video data in an infant care scene, and decompose the video data into multiple frames of images, wherein the infant care scene includes an infant and an infant caregiver; An image segmentation module, used to perform image segmentation on each of the images to determine the position information of the upper limb area of the infant caregiver and the position information of the upper limb area of the infant; A depth analysis module, configured to perform depth analysis on pixel points in the upper limb region of the infant caregiver based on the upper limb region position information of the infant caregiver, and determine foreground region position information and background region position information; The image restoration module is used to fill pixels in the foreground area and the background area respectively according to the foreground area position information and the background area position information, combined with the infant upper limb area position information, to complete the image restoration of the upper limb area of the infant caregiver.
[0059] Specifically, an image segmentation and restoration device for the upper limb region of an infant caregiver provided in an embodiment of the present invention is used, and the device includes: an image decomposition module, used to obtain video data in an infant care scene, and decompose the video data into multiple frames of images, wherein the infant care scene includes an infant and an infant caregiver; an image segmentation module, used to perform image segmentation on each of the images, and determine the position information of the upper limb region of the infant caregiver and the position information of the upper limb region of the infant; a depth analysis module, used to perform depth analysis on pixel points in the upper limb region of the infant caregiver based on the position information of the upper limb region of the infant caregiver, and determine the position information of the foreground region and the position information of the background region; an image restoration module, used to fill pixels in the foreground region and the background region respectively based on the foreground region and the background region, combined with the position information of the upper limb region of the infant, to complete the image restoration of the upper limb region of the infant caregiver. The device decomposes the video data into multiple frames, performs independent image segmentation and analysis on each frame, identifies the upper limb area of the infant caregiver and the infant, and performs deep analysis on the pixel points of the upper limb area of the infant caregiver to accurately distinguish the foreground and background areas. This precise area positioning and deep analysis can dynamically adapt to changes in the scene when processing dynamic videos, reducing interference with other elements in the video. At the same time, based on the position information of the foreground and background, combined with the position information of the infant's upper limb, the foreground and background areas are filled with pixels respectively, while ensuring image restoration, the continuity and naturalness of the video are maintained, avoiding the common image distortion or jump phenomenon in traditional methods, and through meticulous partition processing and frame-by-frame restoration, the details and smoothness of the video can be retained to the maximum extent, making the restored video look more natural and real, thereby improving the visual quality of the video and user experience. In short, when processing dynamic video data, the scheme not only improves the accuracy of image restoration, but also maintains the overall realism and continuity of the video through frame processing, precise segmentation and intelligent filling.
[0060] Example 3
[0061] In addition, combined Figure 1 The method for segmenting and repairing an image of an upper limb region of an infant caregiver described in Embodiment 1 of the present invention may be implemented by an electronic device. Fig. 9 A schematic diagram of the hardware structure of an electronic device provided in Embodiment 3 of the present invention is shown.
[0062] The electronic device may include a processor and a memory storing computer program instructions.
[0063] Specifically, the processor may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiment of the present invention.
[0064] The memory may include a large capacity memory for data or instructions. By way of example and not limitation, the memory may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. Where appropriate, the memory may include a removable or non-removable (or fixed) medium. Where appropriate, the memory may be inside or outside the data processing device. In a particular embodiment, the memory is a non-volatile solid-state memory. In a particular embodiment, the memory includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM) or a flash memory or a combination of two or more of these.
[0065] The processor implements any one of the methods for segmenting and repairing an upper limb area image of an infant caregiver in the above-mentioned embodiments by reading and executing computer program instructions stored in the memory.
[0066] In one example, the electronic device may further include a communication interface and a bus. Fig. 9 As shown, the processor, memory, and communication interface are connected via a bus and communicate with each other.
[0067] The communication interface is mainly used to implement communication between the modules, devices, units and / or equipment in the embodiments of the present invention.
[0068] The bus includes hardware, software or both, and the components of the device are coupled to each other. For example, but not limitation, the bus may include an accelerated graphics port (AGP) or other graphics bus, an enhanced industrial standard architecture (EISA) bus, a front-side bus (FSB), a hypertransport (HT) interconnection, an industrial standard architecture (ISA) bus, an infinite bandwidth interconnection, a low pin count (LPC) bus, a memory bus, a microchannel architecture (MCA) bus, a peripheral component interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a serial advanced technology attachment (SATA) bus, a video electronics standard association local (VLB) bus or other suitable bus or a combination of two or more of these. Where appropriate, the bus may include one or more buses. Although the embodiments of the present invention describe and illustrate a specific bus, the present invention contemplates any suitable bus or interconnection.
[0069] Example 4
[0070] In addition, in combination with the method for segmenting and repairing the upper limb region image of the infant caregiver in the above embodiment 1, embodiment 4 of the present invention can also provide a computer-readable storage medium for implementation. The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by the processor, any of the methods for segmenting and repairing the upper limb region image of the infant caregiver in the above embodiments is implemented.
[0071] In summary, the embodiments of the present invention provide a method, device, equipment and medium for segmenting and repairing images of the upper limb area of an infant caregiver.
[0072] It should be clear that the present invention is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present invention.
[0073] The functional blocks shown in the structural block diagram described above can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier. "Machine-readable medium" may include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROMs, flash memories, erasable ROMs (EROMs), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.
[0074] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant location, and provide corresponding operation entrances for users to choose to authorize or refuse.
[0075] It should also be noted that the exemplary embodiments mentioned in the present invention describe some methods or systems based on a series of steps or devices. However, the present invention is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiments, or in a different order from the embodiments, or several steps can be performed simultaneously.
[0076] The above is only a specific implementation of the present invention. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the system, module and unit described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. It should be understood that the protection scope of the present invention is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be covered within the protection scope of the present invention.
Claims
1. A method for segmenting and repairing an image of an upper limb region of an infant caregiver, characterized in that: The method comprises: S1: Acquire video data of an infant care scene, and decompose the video data into multiple frames of images, wherein the infant care scene includes an infant and an infant caregiver; S2: performing image segmentation on each of the images to determine the position information of the upper limb region of the infant caregiver and the position information of the upper limb region of the infant; S3: performing depth analysis on pixel points in the upper limb area of the infant caregiver according to the upper limb area position information of the infant caregiver to determine the foreground area position information and the background area position information; S4: Based on the foreground area position information and the background area position information, combined with the infant upper limb area position information, the foreground area and the background area are respectively filled with pixels to complete the image restoration of the upper limb area of the infant caregiver.
2. The method for segmenting and repairing an upper limb region image of an infant caregiver according to claim 1, characterized in that: The S2 includes: S21: inputting each of the images into a pre-trained target detection model, and outputting initial position information of the upper limbs of the infant caregiver and initial position information of the upper limbs of the infant; S22: Input each of the images into a pre-trained image segmentation model, perform secondary correction on the infant caregiver upper limb detection frame position information and the infant upper limb detection frame position information, and determine the infant caregiver upper limb region position information and the infant upper limb region position information.
3. The method for segmenting and repairing an upper limb region image of an infant caregiver according to claim 1, characterized in that: The S3 includes: S31: inputting each of the images into a pre-trained depth calculation model to output global depth information; S32: Determine depth information of the upper limb region of the infant caregiver based on the position information of the upper limb region of the infant caregiver and the global depth information; S33: extracting contour features of the upper limb region of the infant caregiver based on the position information of the upper limb region of the infant caregiver, and determining the contour position information of the upper limb region of the infant caregiver; S34: determining the contour depth information of the upper limb area of the infant caregiver according to the depth information of the upper limb area of the infant caregiver and the contour position information of the upper limb area of the infant caregiver; S35: According to preset rules, depth analysis is performed on the contour depth information of the upper limb area of the infant caregiver to determine the background area position information and the foreground area position information.
4. The method for segmenting and repairing an upper limb region image of an infant caregiver according to claim 3, characterized in that: The S35 includes: S351: selecting a pixel point on the contour of the upper limb area of the infant caregiver as the central pixel point, and taking the depth value corresponding to the central pixel point as the reference depth value according to the depth information of the contour of the upper limb area of the infant caregiver; S352: taking depth values of a plurality of target pixel points within a preset range around the central pixel point as target depth values, performing difference calculation between each target depth value and the reference depth value, and determining each depth difference; S353: Classifying the central pixel point as a background point or a foreground point according to each of the depth differences and a preset depth threshold; S354: Repeat steps S351 to S353 for all pixel points on the contour of the upper limb area of the infant caregiver until all pixel points are classified into background points and foreground points; S355: Determine the background area position information and the foreground area position information according to the background point position information and the foreground point position information.
5. The method for segmenting and repairing an upper limb region image of an infant caregiver according to claim 4, characterized in that: The S353 includes: S3531: Compare each depth difference with a preset depth threshold, and determine the proportion of target pixel points whose depth difference is greater than the depth threshold; S3532: If the target pixel ratio is greater than a preset ratio threshold, classify the central pixel as a background point; S3533: If the target pixel ratio is less than or equal to a preset ratio threshold, the central pixel is classified as a foreground point.
6. The method for segmenting and repairing an upper limb region image of an infant caregiver according to claim 5, characterized in that: The S4 includes: S41: judging whether a second pixel point in the upper limb region of the infant is located around a first pixel point in the upper limb region of the infant caregiver according to the upper limb region position information of the infant caregiver and the upper limb region position information of the infant; S42: If the first pixel point is surrounded by the second pixel point, the first pixel point is removed and no pixel filling is performed; S43: If there are no second pixels around the first pixel, perform pixel filling on the first pixel.
7. The method for segmenting and repairing an upper limb region image of an infant caregiver according to claim 5, characterized in that: The S43 includes: S431: If there is no second pixel point around the first pixel point, determine that the first pixel point is a background point or a foreground point, and obtain a preset domain pixel window; S432: If the first pixel is a background point, the color channel average of all pixels in the domain pixel window is used as the color channel value of the first pixel; S433: If the first pixel point is a foreground point, a preset approximate nearest neighbor search algorithm is used to determine a best matching area in the domain pixel window, and the texture information of the best matching area is filled into the first pixel point.
8. An image segmentation and restoration device for the upper limb area of an infant caregiver, characterized in that: The device comprises: An image decomposition module, used to obtain video data in an infant care scene, and decompose the video data into multiple frames of images, wherein the infant care scene includes an infant and an infant caregiver; An image segmentation module, used to perform image segmentation on each of the images to determine the position information of the upper limb area of the infant caregiver and the position information of the upper limb area of the infant; A depth analysis module, configured to perform depth analysis on pixel points in the upper limb region of the infant caregiver based on the upper limb region position information of the infant caregiver, and determine foreground region position information and background region position information; The image restoration module is used to fill pixels in the foreground area and the background area respectively according to the foreground area position information and the background area position information, combined with the infant upper limb area position information, to complete the image restoration of the upper limb area of the infant caregiver.
9. An electronic device, characterized in that: include: At least one processor, at least one memory and computer program instructions stored in the memory, when the computer program instructions are executed by the processor, implement the method according to any one of claims 1 to 7.
10. A storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.
Citation Information
Patent Citations
Image processing method and device, electronic equipment and storage medium
CN112258404A
Image restoration method and device, electronic equipment and computer readable storage medium
CN116167925A
Image restoration method and device, equipment and storage medium
CN118172292A
Method for foreground and background determination in an image
US20190102886A1
Single Image 3D Photography with Soft-Layering and Depth-aware Inpainting
US20240249422A1
