A privacy protection method, apparatus, device and medium

Abstract

The application discloses a privacy protection method and device, equipment and medium, in the application, a color image containing a smart refrigerator and a target user is acquired by a color image acquisition device, a target pixel point corresponding to the target user in the color image is determined, the target pixel point is processed, a target image is acquired and uploaded. Since in the application, before uploading the color image containing the smart refrigerator and the target user, the target pixel point corresponding to the target user in the color image is determined and processed, the target image obtained after processing is uploaded, effectively avoiding the problem of user privacy leakage, and improving the user experience.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a privacy protection method, apparatus, device, and medium. Background Technology

[0002] With the rapid development of information technology, smart refrigerators are generally equipped with image acquisition devices. These devices capture images of both the smart refrigerator and the user, and monitor the user's actions in storing and retrieving food from the refrigerator in real time based on these images, facilitating intelligent management of the food in the refrigerator.

[0003] However, when at home, users dress relatively casually. After collecting images of the smart refrigerator and the user, the images contain the user's human information, which poses a risk of privacy leakage when the images are uploaded to the cloud server for identification, causing great inconvenience to the user. Summary of the Invention

[0004] This application provides a privacy protection method, apparatus, device, and medium to address the risk of privacy leakage when directly uploading images containing smart refrigerators and users to the cloud for identification in the prior art.

[0005] This application provides a privacy protection method, the method comprising:

[0006] Obtain color images of the smart refrigerator and the target user captured by the color image acquisition device;

[0007] Determine the target pixel point corresponding to the target user in the color image;

[0008] The target pixels are processed to obtain the target image and then uploaded.

[0009] Further, the method for determining the target pixel point corresponding to the target user in the color image includes:

[0010] A point cloud image containing the smart refrigerator and the target user is acquired by a depth image acquisition device, wherein the acquisition time of the point cloud image is the same as the acquisition time of the color image;

[0011] Based on the coordinate information corresponding to each point cloud data in the point cloud image and the preset clustering algorithm, the target point cloud data corresponding to the target user is determined;

[0012] Based on the pre-saved correspondence between the pixel coordinate system and the world coordinate system, the target coordinate information corresponding to the target point cloud data is determined in the color image, and the pixel at the target coordinate information is determined as the target pixel.

[0013] Further, determining the target point cloud data corresponding to the target user based on the coordinate information corresponding to each point cloud data in the point cloud image and a preset clustering algorithm includes:

[0014] Based on the coordinate information corresponding to each point cloud data in the point cloud image and the preset clustering algorithm, the candidate point cloud data corresponding to the target user is determined;

[0015] Based on the pre-saved coordinate range corresponding to the user's arm, determine the first point cloud data corresponding to the target user's arm in the candidate point cloud data;

[0016] The other point cloud data in the candidate point cloud data, excluding the first point cloud data, are determined as the target point cloud data corresponding to the target user.

[0017] Further, determining the other point cloud data in the candidate point cloud data besides the first point cloud data as the target point cloud data corresponding to the target user includes:

[0018] Based on the preset and saved coordinate range corresponding to the sensitive body parts of the user, determine the third point cloud data corresponding to the sensitive body parts of the target user in the other point cloud data besides the first point cloud data in the candidate point cloud data;

[0019] A fourth point cloud data is determined within a preset range outside the area where the third point cloud data is located, and the other point cloud data in the candidate point cloud data excluding the first point cloud data and the fourth point cloud data are determined as the target point cloud data corresponding to the target user.

[0020] Further, the processing of the target pixel includes:

[0021] Cut the target pixel; or

[0022] Replace the pixel value of the target pixel.

[0023] This application also provides a privacy protection device, which includes:

[0024] The acquisition module is used to obtain color images of the smart refrigerator and the target user captured by the color image acquisition device;

[0025] The determining module is used to determine the target pixel point corresponding to the target user in the color image;

[0026] The processing module is used to process the target pixels to obtain the target image and upload it.

[0027] Further, the determining module is specifically used to obtain a point cloud image containing the smart refrigerator and the target user acquired by the depth image acquisition device, wherein the acquisition time of the point cloud image is the same as the acquisition time of the color image; determine the target point cloud data corresponding to the target user based on the coordinate information corresponding to each point cloud data in the point cloud image and a preset clustering algorithm; determine the target coordinate information corresponding to the coordinate information of the target point cloud data in the color image based on the pre-saved correspondence between the pixel coordinate system and the world coordinate system, and determine the pixel point at the target coordinate information as the target pixel point.

[0028] Furthermore, the determining module is specifically used to determine candidate point cloud data corresponding to the target user based on the coordinate information corresponding to each point cloud data in the point cloud image and a preset clustering algorithm; to determine the first point cloud data corresponding to the target user's arm in the candidate point cloud data based on the coordinate range corresponding to the user's arm in the pre-saved coordinate range; and to determine the other point cloud data in the candidate point cloud data, excluding the first point cloud data, as the target point cloud data corresponding to the target user.

[0029] Furthermore, the determining module is specifically used to determine, based on the preset coordinate range corresponding to the sensitive body parts of the user, the third point cloud data corresponding to the sensitive body parts of the target user in the other point cloud data besides the first point cloud data in the candidate point cloud data; determine the fourth point cloud data in a preset range outside the area where the third point cloud data is located; and determine the other point cloud data besides the first point cloud data in the candidate point cloud data and the fourth point cloud data as the target point cloud data corresponding to the target user.

[0030] Furthermore, the processing module is specifically used to cut the target pixel; or to replace the pixel value of the target pixel.

[0031] This application also provides an electronic device including a processor, which executes a computer program stored in a memory to implement the steps of any of the privacy protection methods described above.

[0032] This application also provides a computer-readable storage medium storing a computer program executable by a terminal, which, when run on the terminal, causes the terminal to perform the steps of any of the above-described privacy protection methods.

[0033] In this application, a color image containing a smart refrigerator and a target user is obtained from a color image acquisition device. The target pixel corresponding to the target user in the color image is determined, processed, and then the target image is obtained and uploaded. Because this application determines and processes the target pixel corresponding to the target user in the color image before uploading it, the issue of user privacy leakage is effectively avoided, and the user experience is improved. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 A schematic diagram illustrating a privacy protection method provided in some embodiments of this application;

[0036] Figure 2a A schematic diagram illustrating a process for obtaining a target image provided in some embodiments of this application;

[0037] Figure 2b A schematic diagram of a detection process provided for some embodiments of this application;

[0038] Figure 3a This application provides a point cloud image display diagram based on a depth image acquisition device, as shown in some embodiments.

[0039] Figure 3b This application provides a color image display diagram based on a color image acquisition device, as shown in some embodiments.

[0040] Figure 3c A display diagram of a target image obtained according to some embodiments of this application;

[0041] Figure 4a A point cloud image display diagram provided for some embodiments of this application;

[0042] Figure 4b A color image display diagram provided for some embodiments of this application;

[0043] Figure 4c A display diagram of a target image provided for some embodiments of this application;

[0044] Figure 5This application provides a schematic diagram of the structure of a smart refrigerator according to some embodiments;

[0045] Figure 6 A schematic diagram of a privacy protection device structure provided for some embodiments of this application;

[0046] Figure 7 An electronic device provided for some embodiments of this application. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0048] In this application, a color image containing a smart refrigerator and a target user is obtained from a color image acquisition device. The target pixel corresponding to the target user in the color image is determined, processed, and then the target image is obtained and uploaded. Because this application determines and processes the target pixel corresponding to the target user in the color image before uploading it, the issue of user privacy leakage is effectively avoided, and the user experience is improved.

[0049] To protect user privacy and improve user experience, this application provides a privacy protection method, apparatus, device, and medium.

[0050] Figure 1 This application provides a schematic diagram of a privacy protection method according to some embodiments, the process including the following steps:

[0051] S101: Obtain a color image of the smart refrigerator and the target user captured by the color image acquisition device.

[0052] The privacy protection method provided in this application is applied to a smart refrigerator.

[0053] To facilitate the identification of user access to food items within the smart refrigerator, this application pre-installs an image acquisition device within the smart refrigerator to capture images containing both the smart refrigerator and the user. This image acquisition device can be a color image acquisition device. In this application, the smart refrigerator obtains a color image of the smart refrigerator and the target user captured by this color image acquisition device.

[0054] In this application, the smart refrigerator can determine the user's access to food items and identify the types of food items based on the color image. In other words, it can perform food access and type identification locally. If local identification of food access and type fails, the color image can be uploaded to a cloud server, where the cloud server can then perform the food access and type identification. Alternatively, the cloud server can collect sample data, train and optimize the food type identification model and food access operation identification model based on this data, and update the optimized food type identification model and food access operation identification model locally on the smart refrigerator, thereby improving the accuracy of local food type and food access operation identification. Specifically, the process of training and optimizing the food type identification model and food access operation identification model based on sample data is existing technology and will not be elaborated upon here.

[0055] Since there is a risk of privacy leakage when color images containing target users are uploaded to cloud servers, this application mainly focuses on the processing of color images that need to be uploaded to cloud servers. Subsequently, the cloud server performs food storage and retrieval operations and food type identification on the received color images.

[0056] S102: Determine the target pixel point corresponding to the target user in the color image.

[0057] In this application, to protect the information of the target user contained in the color image, the target pixels corresponding to the target user in the color image are first determined. Specifically, a human detection method can be used to outline the human body corresponding to the target user in the color image with a rectangle, and the pixels contained within the rectangle are the target pixels. Alternatively, Gaussian modeling, frame difference extraction of moving human bodies, human body segmentation methods, or human keypoint detection methods can also be used to determine the target pixels corresponding to the target user in the color image.

[0058] The process of determining the target pixel point corresponding to the target user in the color image based on human detection methods, Gaussian modeling, frame difference extraction of moving human body, human body segmentation methods, or human key point detection methods is existing technology and will not be elaborated here.

[0059] S103: Process the target pixels to obtain the target image and upload it.

[0060] After identifying the target pixel corresponding to the target user in the color image, to protect user privacy, this application can process the target pixel to obtain a processed image, which is the target image. After obtaining the target image, it is uploaded, for example, to a cloud server, so that the cloud server can subsequently monitor the user's access to food in the smart refrigerator and identify food types in real time based on the target image. In this application, when processing the target pixel, the pixel value of the target pixel can be processed.

[0061] In this application, after the smart refrigerator uploads the target image to the cloud server, the cloud server, upon receiving the target image, can use two-stage deep learning-based object detection algorithms (Rejion Convolutional Neural Network, RCNN), one-stage object detection algorithms (You Only LookOnce, YOLO), and object detection algorithms (Single Shot MultiBox Detector, SSD) to detect the type of food. The cloud server can also use a series of Siamese-Region Proposal Networks (SiamRPN) algorithms to determine the user's access operations to the food in the smart refrigerator. The process of the cloud server performing food type detection and access action detection based on the target image is existing technology and will not be elaborated upon here.

[0062] In this application, before uploading the color image containing the smart refrigerator and the target user, the target pixel corresponding to the target user in the color image is determined and processed before uploading the processed target image, which effectively avoids the problem of user privacy leakage and improves the user experience.

[0063] To determine the target pixel corresponding to the target user, based on the above embodiments, in this embodiment of the application, the method for determining the target pixel corresponding to the target user in the color image includes:

[0064] A point cloud image containing the smart refrigerator and the target user is acquired by a depth image acquisition device, wherein the acquisition time of the point cloud image is the same as the acquisition time of the color image;

[0065] Based on the coordinate information corresponding to each point cloud data in the point cloud image and the preset clustering algorithm, the target point cloud data corresponding to the target user is determined;

[0066] Based on the pre-saved correspondence between the pixel coordinate system and the world coordinate system, the target coordinate information corresponding to the target point cloud data is determined in the color image, and the pixel at the target coordinate information is determined as the target pixel.

[0067] In this application, the target pixel corresponding to the target user in the color image can be directly identified and processed. In order to improve the accuracy of determining the target pixel corresponding to the target user, a depth image acquisition device can also be pre-installed in the smart refrigerator. The depth image acquisition device is used to acquire point cloud images, wherein the point cloud images contain point cloud data corresponding to the smart refrigerator and the target user.

[0068] To determine the point cloud data corresponding to the target user in the point cloud image, the target point cloud data corresponding to the target user can be determined based on the coordinate information of each point cloud data in the point cloud image and a preset clustering algorithm. The coordinate information corresponding to the point cloud data is three-dimensional coordinate information. The preset clustering algorithm can be mean shift clustering, density-based clustering (DBSCAN), etc.

[0069] In this application, the target point cloud data corresponding to the target user is determined based on the coordinate information of each point cloud data in the point cloud image and the K-means algorithm. Specifically, since the point cloud data corresponding to the target user will be significantly different from the point cloud data corresponding to other non-target users, such as the point cloud data corresponding to the smart refrigerator door, the point cloud data corresponding to the smart refrigerator body, etc., in this application, two group centers can be pre-set. Based on the coordinate information of each point cloud data, the Euclidean distance between each point cloud data and the two pre-set group centers is determined. Then, based on the Euclidean distance, the point cloud data corresponding to the target user is divided into one group, and the point cloud data corresponding to non-target users is divided into another group.

[0070] Clustering point cloud data using the K-means algorithm is an existing technique and will not be elaborated upon here.

[0071] In this application, in order to determine the target pixel corresponding to the target user in a color image, after determining the target point cloud data corresponding to the target user, the target coordinate information corresponding to the target point cloud data in the color image can be determined according to the pre-saved correspondence between the world coordinate system and the pixel coordinate system. The pixel at the target coordinate information is the pixel corresponding to the target user in the color image, and the pixel corresponding to the target user in the color image is the target pixel.

[0072] Specifically, based on the pre-saved correspondence between the world coordinate system and the pixel coordinate system of the color image, as well as the correspondence between the world coordinate system and the pixel coordinate system of the point cloud image, the correspondence between the pixel coordinate system of the color image and the pixel coordinate system of the point cloud image is determined. After determining the coordinate information corresponding to the target point cloud data of the target user in the point cloud image, the target coordinate information corresponding to the target point cloud data of the target user in the point cloud image is determined according to the correspondence between the pixel coordinate system of the color image and the pixel coordinate system of the point cloud image.

[0073] Furthermore, since the coordinate information corresponding to the point cloud data is three-dimensional, the coordinate information corresponding to the point cloud data on the point cloud image can also be directly determined as the coordinate information in the corresponding world coordinate system.

[0074] To accurately determine the target point cloud data corresponding to the target user, based on the above embodiments, in this embodiment, determining the target point cloud data corresponding to the target user according to the coordinate information corresponding to each point cloud data in the point cloud image and a preset clustering algorithm includes:

[0075] Based on the coordinate information corresponding to each point cloud data in the point cloud image and the preset clustering algorithm, the candidate point cloud data corresponding to the target user is determined;

[0076] Based on the pre-saved coordinate range corresponding to the user's arm, determine the first point cloud data corresponding to the user's arm in the candidate point cloud data;

[0077] The other point cloud data in the candidate point cloud data, excluding the first point cloud data, are determined as the target point cloud data corresponding to the target user.

[0078] In this application, all point cloud data corresponding to the target user, determined based on the coordinate information of each point cloud data in the point cloud image and a preset clustering algorithm, can be used as target point cloud data.

[0079] Since the color image containing the smart refrigerator and the target user is mainly used to identify the user's food storage and retrieval operations, it is necessary to ensure that the pixel values ​​of the pixels corresponding to the target food in the color image do not change. In other words, the pixels corresponding to the food in the color image are not processed. Since users handle food with their hands, there's a possibility that the point cloud data corresponding to the user's hand and the food might not be easily distinguishable. Therefore, if all point cloud data corresponding to the user are considered target point cloud data, and the pixels at the corresponding coordinates in the color image are processed, some food pixels might also be processed, affecting subsequent food identification. Because human arms are not considered highly private, pixels at locations other than the user's arm can be processed in the color image. This means identifying all point cloud data corresponding to the user except for the user's arm. All point cloud data corresponding to the user are then considered candidate point cloud data, the point cloud data corresponding to the user's arm is considered the first point cloud data, and the other point cloud data is considered the target point cloud data. Then, the pixels at the corresponding coordinates in the color image are processed.

[0080] In this application, a world coordinate system is pre-set. This world coordinate system uses the midpoint of the top of the smart refrigerator as the origin, a straight line perpendicular to the ground as the z-axis, and a horizontal direction parallel to the outer plane of the refrigerator as the x-axis when the refrigerator door is closed, and a direction perpendicular to the outer plane of the refrigerator as the y-axis. In this application, if the coordinate information corresponding to the point cloud data in the point cloud image is directly identified as the coordinate information in the world coordinate system, since the coordinate information of the candidate point cloud data corresponding to the target user is three-dimensional coordinate information, that is, after determining the candidate point cloud data corresponding to the target user in the point cloud image, the x-axis coordinate information, y-axis coordinate information, and z-axis coordinate information corresponding to each candidate point cloud data are already known.

[0081] Specifically, in determining the first point cloud data corresponding to the target user's arm, since the user's height range is known (typically between 1.2m and 1.9m), the height of the user's arm from the ground is also known (typically between 0.8m and 1.6m). The range of the user's arm's extension is also known. Therefore, to determine the first point cloud data corresponding to the target user's arm, the coordinate range corresponding to the user's arm is pre-saved. Based on this coordinate range, all coordinate information contained within that range can be used to determine the target coordinate information within that range. This target coordinate is the coordinate information corresponding to the first point cloud data.

[0082] Because everyone's height is different, the height at which they raise their arms when accessing food in a smart refrigerator may also vary. This could lead to differences in the coordinate information of the point cloud data corresponding to the arm part when people of different heights or with different arm heights are involved. If all coordinate information within the coordinate range corresponding to the arm part is directly identified as the target coordinate information corresponding to the target user's arm part, and the point cloud data located at the target coordinate information is identified as the first point cloud data, the accuracy of the identified first point cloud data will be low.

[0083] To improve the accuracy of determining the first point cloud data, since the user's arm extends along the y-axis of the coordinate system when storing or retrieving food in the smart refrigerator, the two ends of the point cloud data corresponding to the arm in the point cloud image show the greatest variation in the y-axis direction. In other words, there is a certain difference between the y-coordinate information of the point cloud data corresponding to the two ends of the arm. Therefore, after determining the coordinate information of all point cloud data within the coordinate range corresponding to the arm, the variation of the point cloud data at both ends of the region formed by all point cloud data in each interval can be determined sequentially according to the set range interval length. If the variation is greater than the preset variation threshold, it means that all the point cloud data corresponding to that interval is the first point cloud data corresponding to the arm. For example, if the coordinate range corresponding to the user's arm is 0.8m-1.6m, then the set range interval can be 0.2m. Specifically, the set range interval is set according to the requirements. The amplitude of the change of the point cloud data at both ends of the region formed by all the point cloud data corresponding to each interval in the coordinate range of 0.8m-1m, 1m-1.2m, 1.2m-1.4m, and 1.4m-1.6m in the y-axis direction is determined. Based on the comparison between the amplitude corresponding to each interval and the preset amplitude threshold, the first point cloud data corresponding to the arm is determined.

[0084] To accurately determine the target point cloud data corresponding to the target user, based on the above embodiments, in this embodiment, determining the other point cloud data besides the first point cloud data in the candidate point cloud data as the target point cloud data corresponding to the target user includes:

[0085] Based on the preset and saved coordinate range corresponding to the sensitive body parts of the user, determine the third point cloud data corresponding to the sensitive body parts of the target user in the other point cloud data besides the first point cloud data in the candidate point cloud data;

[0086] A fourth point cloud data is determined within a preset range outside the area where the third point cloud data is located, and the other point cloud data in the candidate point cloud data excluding the first point cloud data and the fourth point cloud data are determined as the target point cloud data corresponding to the target user.

[0087] In this application, because the privacy of certain sensitive body parts of the user is more important, such as the user's chest, buttocks, etc., it is necessary to strengthen the protection of the user's sensitive body parts.

[0088] In this application, similar to the user's arm, since the coordinate range corresponding to the sensitive privacy part is also known, the coordinate range corresponding to the user's sensitive body part is pre-saved in this application. Based on the pre-saved coordinate range corresponding to the user's sensitive body part, the third point cloud data corresponding to the sensitive body part of the target user is determined in the other point cloud data besides the first point cloud data in the candidate point cloud data. For example, the third point cloud data corresponding to the chest is determined, etc.

[0089] After identifying the sensitive body parts of the target user, in order to strengthen the protection of the sensitive body parts, the range of the third point cloud data corresponding to the sensitive body parts can be expanded. For example, other point cloud data around the identified sensitive body parts can also be used as the point cloud data corresponding to the sensitive body parts, and the pixels at the target coordinate information in the color image corresponding to the other point cloud data around the identified sensitive body parts can be processed.

[0090] Specifically, in this application, a fourth point cloud data is determined within a preset range outside the area of ​​the third point cloud data, and the other point cloud data in the candidate point cloud data, excluding the first point cloud data, and the fourth point cloud data are determined as the target point cloud data corresponding to the target user. The preset range is set according to the requirements. The larger the preset range, the more pixels will be processed subsequently.

[0091] Figure 2a This is a schematic diagram illustrating a process for obtaining a target image, provided in some embodiments of this application. Figure 2b This is a schematic diagram of a detection process provided in some embodiments of this application, and is now specifically for... Figure 2a and Figure 2b Please provide an explanation.

[0092] The system can directly acquire color images of the smart refrigerator and the target user from the color image acquisition device in the smart refrigerator, and cache these color images. Since the color image acquisition device can acquire multiple images within a set time period, and each image is a snapshot of a user's food storage and retrieval operation, in order to reduce the workload of subsequent image processing of the color images by the smart refrigerator and the workload of food recognition by the cloud server, frames can be skipped as needed to reduce the amount of color image data that needs to be processed and uploaded.

[0093] After obtaining the color image containing the smart refrigerator and the target user, the human body region of the target user is detected. That is, the target pixel point corresponding to the target user on the color image is determined. Then, the human body region information is processed, that is, the target pixel point corresponding to the target user is processed to obtain the target image and upload it to the cloud server for detection.

[0094] If based on the smart refrigerator itself, the system first detects the user's food storage and retrieval operations. After obtaining a color image containing both the smart refrigerator and the target user, it detects the food type and the location of the storage / retrieval action to determine if the smart refrigerator can detect these parameters locally. If not, it detects the target user's human body area based on the color image, processes this information to obtain a target image, and uploads it to the cloud server. Alternatively, to ensure accuracy, the smart refrigerator first outputs the detection results for user confirmation. If a user correction instruction is received, it detects the target user's human body area based on the color image, processes this information to obtain a target image, and uploads it to the cloud server. If no user correction instruction is received, the detection results are directly output.

[0095] Figure 3a This application provides a point cloud image display diagram based on a depth image acquisition device, as shown in some embodiments. Figure 3b This application provides a color image display diagram based on a color image acquisition device, as shown in some embodiments. Figure 3c This application provides a display diagram of a target image obtained through some embodiments, and now focuses on... Figure 3a , Figure 3b and Figure 3c Please provide an explanation.

[0096] This point cloud image contains multiple point cloud data, such as Figure 3aAs shown. Based on the coordinate information corresponding to each point cloud data in the point cloud image and a preset clustering algorithm, candidate point cloud data corresponding to the target user is determined. Based on the pre-saved coordinate range corresponding to the user's arm, the first point cloud data corresponding to the target user's arm in the candidate point cloud data is determined. Based on the pre-saved coordinate range corresponding to the user's sensitive body parts, the third point cloud data corresponding to the target user's sensitive body parts in the other point cloud data besides the first point cloud data in the candidate point cloud data is determined. A fourth point cloud data within a preset range outside the area where the third point cloud data is located is determined. The other point cloud data besides the first point cloud data and the fourth point cloud data in the candidate point cloud data are then identified as the target point cloud data corresponding to the target user. Then, based on the pre-saved correspondence between the pixel coordinate system and the world coordinate system, the target coordinate information corresponding to the target point cloud data is determined in the color image, and the pixel at the target coordinate information is identified as the target pixel. The target pixel is then processed to obtain the target image, as shown below. Figure 3c As shown.

[0097] Figure 4a This application provides a point cloud image display diagram for some embodiments. Figure 4b This application provides a color image display diagram for some embodiments. Figure 4c This application provides a display diagram of a target image for some embodiments, which is now intended for... Figure 4a , Figure 4b and Figure 4c Please provide an explanation.

[0098] Figure 4a , Figure 4b and Figure 4c The determination and processing process and Figure 3a , Figure 3b and Figure 3c The determination and processing procedures are the same, and will not be repeated here.

[0099] To protect user privacy, based on the above embodiments, in this embodiment, the processing of the target pixel includes:

[0100] Cut the target pixel; or

[0101] Replace the pixel value of the target pixel.

[0102] In this application, in order to protect user privacy, after determining the target pixel corresponding to the target user, the target pixel can be processed to ensure that when the processed target image is uploaded to the cloud server, the information of the target pixel corresponding to the target user cannot be parsed, that is, the information of the target user's human body cannot be parsed.

[0103] Specifically, in this application, when processing a target pixel, the target pixel can be cropped, that is, the area corresponding to the target pixel can be deleted from the color image.

[0104] In this application, the pixel values ​​of the target pixel can also be replaced. Specifically, the pixel values ​​of the target pixel can be replaced with the pixel values ​​corresponding to the default colors, such as the pixel values ​​corresponding to blue, the pixel values ​​corresponding to red, etc., or replaced with a preset pixel value. Furthermore, the target pixel corresponding to the target user can be pixelated.

[0105] Figure 5 This is a structural schematic diagram of a smart refrigerator provided in some embodiments of this application. Now, regarding... Figure 5 Explanation:

[0106] There are two image acquisition devices in the middle of the upper edge of the smart refrigerator: a color image acquisition device for acquiring color images containing the user and the smart refrigerator, and a depth image acquisition device for acquiring point cloud images containing the user and the smart refrigerator.

[0107] Figure 6 A schematic diagram of a privacy protection device provided in some embodiments of this application is shown. The device includes:

[0108] The acquisition module 601 is used to acquire color images of the smart refrigerator and the target user captured by the color image acquisition device.

[0109] The determining module 602 is used to determine the target pixel point corresponding to the target user in the color image;

[0110] The processing module 603 is used to process the target pixels to obtain a target image and upload it.

[0111] In one possible implementation, the determining module 602 is specifically used to obtain a point cloud image containing the smart refrigerator and the target user, acquired by a depth image acquisition device, wherein the acquisition time of the point cloud image is the same as the acquisition time of the color image; determine the target point cloud data corresponding to the target user based on the coordinate information corresponding to each point cloud data in the point cloud image and a preset clustering algorithm; determine the target coordinate information corresponding to the coordinate information of the target point cloud data in the color image based on the pre-saved correspondence between the pixel coordinate system and the world coordinate system, and determine the pixel point at the target coordinate information as the target pixel point.

[0112] In one possible implementation, the determining module 602 is specifically used to determine candidate point cloud data corresponding to the target user based on the coordinate information corresponding to each point cloud data in the point cloud image and a preset clustering algorithm; determine the first point cloud data corresponding to the target user's arm in the candidate point cloud data based on the coordinate range corresponding to the user's arm in the pre-saved coordinate range; and determine the other point cloud data in the candidate point cloud data other than the first point cloud data as the target point cloud data corresponding to the target user.

[0113] In one possible implementation, the determining module 602 is specifically configured to determine, based on the preset coordinate range corresponding to the sensitive body parts of the user, the third point cloud data corresponding to the sensitive body parts of the target user in the other point cloud data besides the first point cloud data in the candidate point cloud data; determine the fourth point cloud data in a preset range outside the area where the third point cloud data is located; and determine the other point cloud data besides the first point cloud data in the candidate point cloud data and the fourth point cloud data as the target point cloud data corresponding to the target user.

[0114] In one possible implementation, the processing module 603 is specifically used to cut the target pixel; or to replace the pixel value of the target pixel.

[0115] Based on the above embodiments, some embodiments of this application also provide an electronic device, such as... Figure 7 As shown, it includes: processor 701, communication interface 702, memory 703 and communication bus 704, wherein processor 701, communication interface 702 and memory 703 communicate with each other through communication bus 704.

[0116] The memory 703 stores a computer program, which, when executed by the processor 701, causes the processor 701 to perform the following steps:

[0117] Obtain color images of the smart refrigerator and the target user captured by the color image acquisition device;

[0118] Determine the target pixel point corresponding to the target user in the color image;

[0119] The target pixels are processed to obtain the target image and then uploaded.

[0120] Furthermore, the processor 701 is also configured to obtain a point cloud image containing the smart refrigerator and the target user acquired by the depth image acquisition device, wherein the acquisition time of the point cloud image is the same as the acquisition time of the color image; determine the target point cloud data corresponding to the target user based on the coordinate information corresponding to each point cloud data in the point cloud image and a preset clustering algorithm; determine the target coordinate information corresponding to the coordinate information of the target point cloud data in the color image based on the pre-saved correspondence between the pixel coordinate system and the world coordinate system, and determine the pixel point at the target coordinate information as the target pixel point.

[0121] Furthermore, the processor 701 is also configured to determine candidate point cloud data corresponding to the target user based on the coordinate information corresponding to each point cloud data in the point cloud image and a preset clustering algorithm; determine the first point cloud data corresponding to the target user's arm in the candidate point cloud data based on the coordinate range corresponding to the user's arm in the preset point cloud data; and determine the other point cloud data in the candidate point cloud data other than the first point cloud data as the target point cloud data corresponding to the target user.

[0122] Furthermore, the processor 701 is also configured to determine, based on the preset coordinate range corresponding to the sensitive body parts of the user, a third point cloud data corresponding to the sensitive body parts of the target user in the other point cloud data besides the first point cloud data in the candidate point cloud data; determine a fourth point cloud data in a preset range outside the area where the third point cloud data is located; and determine the other point cloud data besides the first point cloud data in the candidate point cloud data and the fourth point cloud data as the target point cloud data corresponding to the target user.

[0123] Furthermore, the processor 701 is also used to cut the target pixel; or to replace the pixel value of the target pixel.

[0124] The communication bus mentioned in the above server can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0125] The communication interface 702 is used for communication between the above-mentioned electronic device and other devices.

[0126] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0127] The processors mentioned above can be general-purpose processors, including central processing units, network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits, field-programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0128] Based on the above embodiments, this application also provides a computer-readable storage medium storing a computer program executable by an electronic device. When the program is run on the electronic device, the electronic device performs the following steps:

[0129] Obtain color images of the smart refrigerator and the target user captured by the color image acquisition device;

[0130] Determine the target pixel point corresponding to the target user in the color image;

[0131] The target pixels are processed to obtain the target image and then uploaded.

[0132] Further, the method for determining the target pixel point corresponding to the target user in the color image includes:

[0133] A point cloud image containing the smart refrigerator and the target user is acquired by a depth image acquisition device, wherein the acquisition time of the point cloud image is the same as the acquisition time of the color image;

[0134] Based on the coordinate information corresponding to each point cloud data in the point cloud image and the preset clustering algorithm, the target point cloud data corresponding to the target user is determined;

[0135] Based on the pre-saved correspondence between the pixel coordinate system and the world coordinate system, the target coordinate information corresponding to the target point cloud data is determined in the color image, and the pixel at the target coordinate information is determined as the target pixel.

[0136] Further, determining the target point cloud data corresponding to the target user based on the coordinate information corresponding to each point cloud data in the point cloud image and a preset clustering algorithm includes:

[0137] Based on the coordinate information corresponding to each point cloud data in the point cloud image and the preset clustering algorithm, the candidate point cloud data corresponding to the target user is determined;

[0138] Based on the pre-saved coordinate range corresponding to the user's arm, determine the first point cloud data corresponding to the target user's arm in the candidate point cloud data;

[0139] The other point cloud data in the candidate point cloud data, excluding the first point cloud data, are determined as the target point cloud data corresponding to the target user.

[0140] Further, determining the other point cloud data in the candidate point cloud data besides the first point cloud data as the target point cloud data corresponding to the target user includes:

[0141] Based on the preset and saved coordinate range corresponding to the sensitive body parts of the user, determine the third point cloud data corresponding to the sensitive body parts of the target user in the other point cloud data besides the first point cloud data in the candidate point cloud data;

[0142] A fourth point cloud data is determined within a preset range outside the area where the third point cloud data is located, and the other point cloud data in the candidate point cloud data excluding the first point cloud data and the fourth point cloud data are determined as the target point cloud data corresponding to the target user.

[0143] Further, the processing of the target pixel includes:

[0144] Cut the target pixel; or

[0145] Replace the pixel value of the target pixel.

[0146] In this application, before uploading the color image containing the smart refrigerator and the target user, the target pixel corresponding to the target user in the color image is determined and processed before uploading the processed target image, which effectively avoids the problem of user privacy leakage and improves the user experience.

[0147] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0148] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0149] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0150] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0151] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A privacy protection method, characterized in that, The method includes: Obtain color images of the smart refrigerator and the target user captured by the color image acquisition device; A point cloud image containing the smart refrigerator and the target user is acquired by a depth image acquisition device, wherein the acquisition time of the point cloud image is the same as the acquisition time of the color image; Based on the coordinate information corresponding to each point cloud data in the point cloud image and the preset clustering algorithm, the target point cloud data corresponding to the target user is determined; Based on the pre-saved correspondence between the pixel coordinate system and the world coordinate system, the target coordinate information corresponding to the target point cloud data is determined in the color image, and the pixel at the target coordinate information is determined as the target pixel. The target pixels are processed to obtain the target image and then uploaded. The step of determining the target point cloud data corresponding to the target user based on the coordinate information corresponding to each point cloud data in the point cloud image and a preset clustering algorithm includes: Based on the coordinate information corresponding to each point cloud data in the point cloud image and the preset clustering algorithm, the candidate point cloud data corresponding to the target user is determined; Based on the pre-saved coordinate range corresponding to the user's arm, the coordinate information of all point cloud data contained within the coordinate range is determined. Then, according to the set range interval length, the amplitude of the change of the point cloud data at both ends of the region formed by all point cloud data corresponding to each interval in the y-axis direction is determined. If the amplitude is greater than the preset amplitude threshold, then all point cloud data corresponding to that interval is determined as the first point cloud data corresponding to the arm. The y-axis direction is the direction perpendicular to the outer plane of the refrigerator. The other point cloud data in the candidate point cloud data, excluding the first point cloud data, are determined as the target point cloud data corresponding to the target user.

2. The method according to claim 1, characterized in that, The step of determining the other point cloud data in the candidate point cloud data, excluding the first point cloud data, as the target point cloud data corresponding to the target user includes: Based on the preset and saved coordinate range corresponding to the sensitive body parts of the user, determine the third point cloud data corresponding to the sensitive body parts of the target user in the other point cloud data besides the first point cloud data in the candidate point cloud data; A fourth point cloud data is determined within a preset range outside the area where the third point cloud data is located, and the other point cloud data in the candidate point cloud data excluding the first point cloud data and the fourth point cloud data are determined as the target point cloud data corresponding to the target user.

3. The method according to claim 1, characterized in that, The processing of the target pixel includes: Cut the target pixel; or Replace the pixel value of the target pixel.

4. A privacy protection device, characterized in that, The device includes: The acquisition module is used to obtain color images of the smart refrigerator and the target user captured by the color image acquisition device; The determination module is used to obtain a point cloud image containing the smart refrigerator and the target user, acquired by a depth image acquisition device, wherein the acquisition time of the point cloud image is the same as the acquisition time of the color image; determine the target point cloud data corresponding to the target user based on the coordinate information corresponding to each point cloud data in the point cloud image and a preset clustering algorithm; determine the target coordinate information corresponding to the coordinate information of the target point cloud data in the color image based on the pre-saved correspondence between the pixel coordinate system and the world coordinate system, and determine the pixel point at the target coordinate information as the target pixel point; The processing module is used to process the target pixels to obtain the target image and upload it. The determining module is specifically used to determine candidate point cloud data corresponding to the target user based on the coordinate information corresponding to each point cloud data in the point cloud image and a preset clustering algorithm; determine the coordinate information of all point cloud data contained within the pre-saved coordinate range corresponding to the user's arm; determine the amplitude of the change of the point cloud data at both ends of the region formed by all point cloud data corresponding to each interval in the y-axis direction according to the set range interval length; if the amplitude is greater than a preset amplitude threshold, then determine all point cloud data corresponding to that interval as the first point cloud data corresponding to the arm, wherein the y-axis direction is perpendicular to the outer plane of the refrigerator; and determine the other point cloud data in the candidate point cloud data other than the first point cloud data as the target point cloud data corresponding to the target user.

5. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory being used to store program instructions, and the processor being used to implement the steps of the privacy protection method according to any one of claims 1-3 when executing the computer program stored in the memory.

6. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the steps of the privacy protection method described in any one of claims 1-3.

Images