Data processing method, electronic device and non-transient computer readable storage medium
By using the privacy content identification module in electronic devices to identify and process the privacy content in keyframes, the problem of the impact of private information in visual-based synchronous positioning technology is solved, and more efficient positioning and map updates are achieved, reducing the computing complexity and information leakage risks.
Patent Information
- Application Number
- CN202410246146.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-10-03
- Filing Date
- 2024-03-05
- Publication Date
- 2025-04-04
AI Technical Summary
In the field of spatial positioning and tracking, visual-based synchronous positioning and map construction technology is easy to obtain private information, resulting in a risk of degradation in computing performance and information leakage.
By introducing a privacy content identification module into the electronic device, the privacy content in the keyframe is identified and image-processed to generate corrected keyframes, thereby updating the environment map and avoiding the use of privacy information for positioning and map updates.
It effectively eliminates the impact of privacy content on location and map updates, reduces the complexity and burden of computing, avoids the risk of information leakage, and improves the performance of electronic devices.
Smart Images
Figure CN119766983A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to a method and an apparatus, and more particularly to a data processing method and an electronic apparatus. Background Art
[0002] In the field of spatial positioning and tracking, devices that use vision-based simultaneous positioning and mapping technology will inevitably obtain any type of private information, such as people, files, etc. This private information is not only unhelpful for spatial positioning and tracking, but also affects the computing performance of the device. Therefore, it is necessary to provide a new method to solve the above problems caused by private information. Summary of the invention
[0003] One aspect of the present disclosure is a data processing method. The data processing method is applicable to an electronic device in an environment and includes: obtaining a key frame; processing the key frame to generate a modified key frame when the key frame includes at least one private content, wherein the modified key frame includes at least one image processing area corresponding to the at least one private content; and using the modified key frame to update a map of the environment.
[0004] In some embodiments, the data processing method further includes: performing image recognition on the key frame by a privacy content recognition module to determine whether the key frame includes the at least one privacy content.
[0005] In some embodiments, the data processing method further includes: analyzing the modified key frame to determine whether the number of multiple first feature points of the modified key frame is greater than a first threshold, wherein when the number of multiple first feature points of the modified key frame is greater than the first threshold, performing an operation of using the modified key frame to update the map of the environment.
[0006] In some embodiments, the operation of analyzing the modified key frame includes: extracting the plurality of first feature points from the modified key frame; calculating the number of the plurality of first feature points; and comparing the number of the plurality of first feature points with the first threshold.
[0007] In some embodiments, the data processing method further includes: obtaining another key frame when the number of the plurality of first feature points of the modified key frame is not greater than the first threshold.
[0008] In some embodiments, the operation of obtaining the other key frame includes: extracting multiple second feature points from an image frame; matching the multiple second feature points with multiple map points of the map to obtain multiple common feature points; and determining the image frame to be the other key frame when the number of the multiple common feature points is not greater than a second threshold.
[0009] In some embodiments, after obtaining the another key frame, the data processing method further includes: performing image recognition on the another key frame by a privacy content recognition module.
[0010] In some embodiments, the plurality of first feature points are not extracted from the at least one image processing region.
[0011] In some embodiments, the data processing method further includes: when the key frame does not include the at least one private content, using the key frame to update the map of the environment.
[0012] In some embodiments, the operation of processing the key frame to generate the modified key frame includes: performing image smoothing or image blurring on the at least one private content to generate the at least one image processing area.
[0013] Another aspect of the present disclosure is an electronic device. The electronic device includes an image capture circuit and a processor. The image capture circuit is used to capture multiple image frames. The processor is coupled to the image capture circuit and is used to: set one of the multiple image frames as a key frame; when the key frame includes at least one private content, process the key frame to generate a modified key frame, wherein the modified key frame includes at least one image processing area corresponding to the at least one private content; and use the modified key frame to update a map of an environment where the electronic device is located.
[0014] Another aspect of the present disclosure is a non-transitory computer-readable storage medium having a computer program for executing a data processing method, wherein the data processing method is applicable to an electronic device in an environment and includes: obtaining a key frame; processing the key frame to generate a modified key frame when the key frame includes at least one private content, wherein the modified key frame includes at least one image processing area corresponding to the at least one private content; and using the modified key frame to update a map of the environment.
[0015] In summary, by eliminating the features of the privacy content in the keyframe, the operation of the processor (e.g., generating and / or updating a map, calculating the position and / or posture of the electronic device, etc.) will not be affected by the privacy information accidentally obtained from the environment, which will reduce the computational complexity and burden of the processor. In addition, in the case where the electronic device must share information related to the environment (e.g., the corrected keyframe) with other devices so that other devices can also locate themselves in the environment, the processor does not need to perform encryption. Therefore, the electronic device and data processing method of the present disclosure have advantages such as improved performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] Figure 1 It is a schematic diagram of an environment where an electronic device is located according to some embodiments of the present disclosure.
[0017] Figure 2 A block diagram of an electronic device according to some embodiments of the present disclosure.
[0018] Figure 3 The present invention is a flowchart of a data processing method according to some embodiments of the present disclosure.
[0019] Figure 4 Schematic diagram of an image frame captured by an electronic device according to some embodiments of the present disclosure.
[0020] Figure 5 Schematic diagram of a modified key frame generated by an electronic device according to some embodiments of the present disclosure.
[0021] Figure 6 It is a schematic diagram of another image frame captured by an electronic device according to some embodiments of the present disclosure.
[0022] 100: Electronic devices
[0023] 201: Image capture circuit
[0024] 203: Processor
[0025] 213: Private content identification module
[0026] 300:Data processing method
[0027] E1: Environment
[0028] F1[1]~F1
[11] : First feature point
[0029] F2[1]~F2[7]: Second feature point
[0030] FC[1]~FC[3]: common feature points
[0031] Img,Img[1],Img[2]: Image frames
[0032] Imgk[1],Imgk[2]: keyframes
[0033] Imgk[1]': Corrected key frame
[0034] M1: Map
[0035] O1: Display
[0036] PA[1],PA[2]: Image processing area
[0037] PI[1],PI[2]: Private content
[0038] S301~S307: Steps
[0039] U1: User
[0040] U2: People
[0041] V1, V2: Field of view DETAILED DESCRIPTION
[0042] The following embodiments are described in detail with reference to the accompanying drawings, but the specific embodiments described are only used to explain the present case and are not used to limit the present case. The description of the structural operation is not used to limit the order of its execution. Any structure recombined by the elements to produce a device with equal functions is within the scope of the present disclosure.
[0043] As used herein, “coupled” or “connected” may refer to two or more elements being in direct physical or electrical contact with each other, or being in indirect physical or electrical contact with each other, or may refer to two or more elements operating or moving with each other.
[0044] See also Figure 1 , Figure 1 FIG. 1 is a schematic diagram of an environment E1 in which an electronic device 100 is located according to some embodiments of the present disclosure. In some embodiments, the electronic device 100 is a head-mounted device. Figure 1 As shown, the electronic device 100 can be installed on the head of a user U1 in an environment E1 such as a public place, a game place, a workplace, etc.
[0045] In some embodiments, the electronic device 100 may provide an immersive content (not shown) for the user U1. The electronic device 100 may block the user U1's direct visibility of the environment E1, and the immersive content may be a virtual reality (VR) environment or a mixed reality (MR) environment. Specifically, the mixed reality environment simulates the environment E1 and enables virtual reality objects (which cannot be directly seen by the user U1 in the environment E1) to interact with the simulated environment. However, the present disclosure is not limited to this. For example, the electronic device 100 may not block the user U1's direct visibility of the environment E1, and the immersive content may be an augmented reality (AR) environment. Specifically, the augmented reality environment enhances the environment E1 directly seen by the user U1 with virtual reality objects.
[0046] Following the above embodiment, the user U1 may control the virtual reality objects in the immersive content by operating at least one controller (not shown) or by making hand or eye movements. Specifically, at least one controller may be wirelessly connected to the electronic device 100, and the hand or eye movements will be sensed and recognized by the electronic device 100.
[0047] It should be understood that the electronic device 100 is not limited to being a head-mounted device. For example, in some embodiments, the electronic device 100 may be an autonomous mobile robot, such as a cleaning robot, a sweeping robot, a food delivery robot, etc. In short, the electronic device 100 may be a device capable of locating itself in the environment E1.
[0048] See also Figure 2 , Figure 2 FIG. 1 is a block diagram of an electronic device 100 according to some embodiments of the present disclosure. Figure 2 As shown, the electronic device 100 includes an image capture circuit 201 and a processor 203. The processor 203 is electrically coupled to the image capture circuit 201.
[0049] In some embodiments, the image capture circuit 201 is used to capture the image frame Img. Specifically, the image capture circuit 201 can be implemented by at least one camera device or a structure including multiple components such as a lens, an image sensor, and an image processor.
[0050] In some embodiments, the processor 203 is used to locate the electronic device 100 in the environment E1 through some vision-based positioning technologies (e.g., simultaneous positioning and mapping technologies, etc.). Through those vision-based positioning technologies, the processor 203 uses the image frames Img received from the image capture circuit 201 to generate and / or update a map M1 of the environment E1 and calculate the position and / or posture of the electronic device 100 in the map M1. Specifically, the processor 203 can be implemented by a central processing unit, a graphics processing unit, an application specific integrated circuit, a microprocessor, a system on a chip, or other suitable processing circuits.
[0051] Because of those vision-based positioning technologies, the electronic device 100 may accidentally obtain some private information from the environment E1. Figure 1 In an embodiment, the private information may be, for example, a portrait of a person U2 in the environment E1, visual content displayed by a display O1 in the environment E1, etc. The private information may be carried by some image frames Img. If the processor 203 uses these image frames Img carrying private information to generate and / or update the map M1 and calculate the position and / or posture of the electronic device 100, the computational complexity and burden of the processor 203 may increase. For example, the person U2 in the environment E1 may be more conspicuous in the environment E1 and / or may have unpredictable movements, which will affect the image feature extraction. In addition, when the electronic device 100 wants to share information with other devices, the electronic device 100 must be further encrypted to prevent the private information from being leaked when these image frames Img carrying private information are to be shared. It should be understood that private information can be referred to as confidential information or sensitive information.
[0052] In view of the above problems caused by privacy information, such as Figure 2As shown, the processor 203 is used to use a privacy content recognition module 213 to perform image recognition on some image frames Img. In some embodiments, the privacy content recognition module 213 is a neural network model that has been well trained and can recognize privacy content from images (for example, a convolutional neural network model, a multi-layer perceptron (MLP) model, etc.). Based on the results of image recognition, the processor 203 can selectively process and analyze the image frames Img that have been recognized. Through these operations, the processor 203 can perform further operations (for example, generate and / or update the map M1, calculate the position and / or posture of the electronic device 100, etc.) without using privacy information. Therefore, the electronic device 100 can avoid the above-mentioned problems. The operation of the processor 203 will be described in the following paragraphs. Figure 3 Detailed description.
[0053] In some embodiments, a training dataset may be first used to adjust multiple parameters of the private content identification module 213. The training dataset consists of multiple training images, each of which includes at least one label indicating the private content in the training image. Next, a verification dataset may be used to adjust multiple hyperparameters of the private content identification module 213. After the private content identification module 213 is well adjusted by the training dataset and the verification dataset, a test dataset may be used to evaluate the performance of the private content identification module 213. In this way, the private content identification module 213 can learn to identify private content from images. In addition, the training of the private content identification module 213 can be completed before the private content identification module 213 is officially used.
[0054] In some embodiments, the electronic device 100 may further include a storage circuit (not shown), wherein the storage circuit may be implemented by a volatile memory, a non-volatile memory, or both. The map M1, the private content identification module 213, and the image frame Img captured by the image capture circuit 201 may be stored in the storage circuit and may be accessed by the processor 203.
[0055] See also Figure 3 , Figure 3 FIG. 3 is a flow chart of a data processing method 300 according to some embodiments of the present disclosure. The data processing method 300 may be executed by the processor 203 of the electronic device 100. In some embodiments, Figure 3 As shown, the data processing method 300 includes a plurality of steps S301 to S307. However, the present disclosure is not limited thereto.
[0056] In step S301, the processor 203 obtains a key frame, which is matched with Figure 1 and 4 The details are then explained. Figure 4 FIG. 1 is a schematic diagram of an image frame Img[1] among a plurality of image frames Img captured by the image capture circuit 201 according to some embodiments of the present disclosure. The image frame Img[1] captured by the image capture circuit 201 may correspond to Figure 1 In some embodiments, the image frame Img[1] is separated from the last / latest key frame (not shown) by a preset number of frames or more, and the number of common feature points between the image frame Img[1] and the last key frame is equal to or less than a preset threshold. In such a case, Figure 4 As shown, the processor 203 determines the image frame Img[1] as a new key frame Imgk[1].
[0057] After obtaining the key frame Imgk[1], step S302 is executed. In step S302, the processor 203 performs image recognition on the key frame Imgk[1]. In some embodiments, the processor 203 performs image recognition on the key frame Imgk[1]. Figure 2 The privacy content identification module 213 identifies the key frame Imgk[1]. The result is as follows Figure 4 As shown, a private content PI[1] and a private content PI[2] are identified from the key frame Imgk[1]. Specifically, the private content PI[1] corresponds to Figure 1 The portrait of person U2, while the private content PI[2] corresponds to Figure 1 The visual content displayed by the display O1.
[0058] Based on the result that the key frame Imgk[1] contains the private content PI[1] and the private content PI[2], the processor 203 must further process and analyze the key frame Imgk[1]. Therefore, step S303 is executed. In step S303, the processor 203 processes the key frame Imgk[1] to generate a modified key frame Imgk[1]', which will be used with Figure 4 and 5 The details are then explained. Figure 5 FIG. 1 is a schematic diagram of a modified key frame Imgk[1]′ according to some embodiments of the present disclosure. In some embodiments, the processor 203 Figure 4 The privacy content PI[1] and the privacy content PI[2] in the image are smoothed (or blurred) to produce Figure 5 In other words, the modified key frame Imgk[1]' includes the image processing area PA[1] and the image processing area PA[2]. Figure 4 and 5 As shown, the image processing area PA[1] corresponds to the privacy content PI[1], and the image processing area PA[2] corresponds to the privacy content PI[2].
[0059] It should be understood that the number of privacy content PI or the number of image processing areas PA is not limited thereto. For example, in some embodiments, at least one privacy content PI is identified from a key frame, and at least one image processing area PA is generated by processing at least one privacy content PI.
[0060] After the modified key frame Imgk[1]' is generated, step S304 is performed. In step S304, the processor 203 analyzes the modified key frame Imgk[1]'. In some embodiments, Figure 5 As shown, the processor 203 extracts a plurality of first feature points F1[1] to F1
[11] from the modified key frame Imgk[1]', calculates the number of the plurality of first feature points F1[1] to F1
[11] , and compares the number of the plurality of first feature points F1[1] to F1
[11] with another preset threshold. As can be seen from the above description, the processor 203 analyzes the modified key frame to determine whether the number of the plurality of first feature points is greater than a first threshold (i.e., the other preset threshold).
[0061] It is worth noting that in some embodiments, because image smoothing eliminates the features of the privacy content PI[1] and PI[2], the processor 203 cannot extract any feature points from the image processing area PA[1] and the image processing area PA[2] generated by image smoothing. Figure 5 As shown, the first feature points F1[1] to F1
[11] are not extracted from the image processing areas PA[1] and PA[2]. It should be understood that image smoothing can be replaced by any suitable image processing technology that can eliminate the features of the privacy content.
[0062] In some embodiments, the processor 203 determines that the number of the first feature points of the modified key frame is greater than the first threshold, so that step S305 is executed. In step S305, the processor 203 uses the modified key frame Imgk[1]' to update the map M1 of the environment E1. It should be understood that the update of the map M1 is well known to those with ordinary knowledge in the technical field of the present disclosure, so its description is omitted here. In addition, in some embodiments, such as Figure 3 As shown, step S301 will be executed again after step S305 is executed.
[0063] In some embodiments, the processor 203 determines that the number of the first feature points of the modified key frame is not greater than the first threshold, and thus the modified key frame is invalidated. A modified key frame without sufficient feature points will be invalidated because it cannot provide sufficient information to update the map M1 (and / or calculate the position and / or posture of the electronic device 100 in the map M1). In some further embodiments, a key frame containing at least one private content PI (e.g., Figure 4 The key frame Imgk[1] shown in FIG. 1 is also invalidated due to at least one privacy content PI. Specifically, invalidating the modified key frame (or key frame) means that the processor 203 may not store the modified key frame (or key frame) in the storage circuit of the electronic device 100. Accordingly, step S306 is executed.
[0064] In step S306, the processor 203 obtains another key frame, which will be matched with Figure 1 and 6 The following is a detailed description. Figure 6 , Figure 6 FIG. 2 is a schematic diagram of another image frame Img[2] among the multiple image frames Img captured by the image capture circuit 201 according to some embodiments of the present disclosure. The image frame Img[2] captured by the image capture circuit 201 may correspond to Figure 1 In some embodiments, the image frame Img[2] is also separated from the last key frame by a predetermined number of frames or more. Figure 6 As shown, the processor 203 extracts multiple second feature points F2[1]~F2[7] from the image frame Img[2], and matches the multiple second feature points F2[1]~F2[7] with multiple map points (not shown) in the map M1 corresponding to the last key frame. In some embodiments, three second feature points F2[1]~F2[3] are found to match and can therefore be referred to as common feature points FC[1]~FC[3]. In other words, the processor 203 obtains multiple common feature points FC[1]~FC[3] between the image frame Img[2] and the last key frame. Thereafter, the processor 203 determines whether the number of the multiple common feature points FC[1]~FC[3] is equal to or less than the preset threshold. Figure 6 In the embodiment of the present invention, the number of the plurality of common feature points FC[1]-FC[3] is equal to or less than the preset threshold, so that the image frame Img[2] is set as a new key frame Imgk[2]. As can be seen from the above description, when the number of the plurality of common feature points is not greater than a second threshold (i.e., the preset threshold), the processor 203 determines that the image frame Img[2] is a key frame Imgk[2].
[0065] In some embodiments, such as Figure 3As shown, step S302 will be executed again after the new key frame Imgk[2] is set. In some further embodiments, the result of image recognition of the key frame Imgk[2] shows that the key frame Imgk[2] does not contain any privacy content. Therefore, the processor 203 does not need to process and analyze the key frame Imgk[2], and step S307 is executed accordingly. In step S307, the processor 203 uses the key frame Imgk[2] to update the map M1 of the environment E1. The update of the map M1 is well known to those having ordinary knowledge in the technical field to which the contents of the present disclosure belong, so its description is omitted here.
[0066] As can be seen from the description of the above data processing method 300, in some embodiments, the data processing method 300 of the present disclosure includes the steps of obtaining a key frame Imgk[1], processing the key frame Imgk[1] to generate a modified key frame Imgk[1]' when the key frame Imgk[1] includes at least one private content (e.g., private content PI[1], private content PI[2], etc.), and using the modified key frame Imgk[1]' to update the map M1 of the environment E1. In some further embodiments, when the number of the plurality of first feature points F1[1] to F1
[11] of the modified key frame Imgk[1]' is greater than a first threshold, the step of using the modified key frame Imgk[1]' to update the map M1 of the environment E1 is performed.
[0067] In the above embodiment, for those image frames Img that are not set as key frames, the processor 203 simply estimates the position and / or posture of the electronic device 100 using only the relative relationship between those image frames Img and the last key frame. Thereafter, those image frames Img may not be retained, that is, those image frames Img may not be stored in the storage circuit of the electronic device 100. Therefore, in some embodiments, even if those image frames Img carry privacy information, the processor 203 will not perform image smoothing on those image frames Img.
[0068] In the above embodiment, if Figure 1 , 4 As shown in FIG. 5 , the processor 203 will Figure 1 The portrait of person U2 is identified as private content PI[1]. However, the present disclosure is not limited thereto. For example, in some embodiments, the processor 203 may identify the complete image of person U2 (i.e., from head to toe) as private content PI. In this case, the processor 203 may perform image smoothing on the complete image of person U2 to generate an image processing area PA accordingly.
[0069] It can be seen from the above embodiments of the present disclosure that by eliminating the features of the privacy content in the key frame, the operation of the processor 203 (e.g., generating and / or updating the map M1, calculating the position and / or posture of the electronic device 100, etc.) will not be affected by the privacy information accidentally obtained from the environment E1, which will reduce the computational complexity and burden of the processor 203. In addition, in the case where the electronic device 100 must share information related to the environment E1 (e.g., the corrected key frame) with other devices so that other devices can also locate themselves in the environment E1, the processor 203 does not need to perform encryption. In summary, the electronic device 100 and the data processing method 300 of the present disclosure have advantages such as improved performance.
[0070] The method of the present disclosure may exist in the form of program code. The program code may be contained in a physical medium, such as a floppy disk, a CD, a hard disk, or any other transient or non-transient computer-readable storage medium, wherein when the program code is loaded and executed by a computer, the computer becomes a device for implementing the method. The program code may also be transmitted through some transmission medium, such as wires or cables, through optical fibers, or through any other transmission mode, wherein when the program code is received, loaded and executed by a computer, the computer becomes a device for implementing the method. When implemented on a general-purpose processor, the program code combines with the processor to provide a unique device that operates similarly to an application-specific logic circuit.
[0071] Although the present disclosure has been disclosed in the above-mentioned implementation mode, it is not intended to limit the present disclosure. A person skilled in the art may make various changes and modifications without departing from the spirit and scope of the present disclosure. Therefore, the protection scope of the present disclosure shall be determined by the following claims.
Claims
1. A data processing method, characterized in that: An electronic device suitable for use in an environment, and comprising: Get a key frame; In the case where the key frame includes at least one private content, processing the key frame to generate a modified key frame, wherein the modified key frame includes at least one image processing area corresponding to the at least one private content; as well as A map of the environment is updated using the revised keyframes.
2. The data processing method according to claim 1, characterized in that: Also includes: The key frame is image-recognized by a privacy content recognition module to determine whether the key frame contains the at least one privacy content.
3. The data processing method according to claim 1, wherein: Also includes: Analyzing the modified key frame to determine whether the number of the plurality of first feature points of the modified key frame is greater than a first threshold, Wherein, when the number of the plurality of first feature points of the modified key frame is greater than the first threshold, an operation of updating the map of the environment using the modified key frame is performed.
4. The data processing method according to claim 3, characterized in that: The operation of analyzing the modified key frame includes: Extracting the plurality of first feature points from the modified key frame; Calculating the number of the plurality of first feature points; and The number of the plurality of first feature points is compared with the first threshold.
5. The data processing method according to claim 3, characterized in that: Also includes: When the number of the plurality of first feature points of the corrected key frame is not greater than the first threshold, another key frame is obtained.
6. The data processing method according to claim 5, characterized in that: The operation of obtaining the other key frame includes: Extracting a plurality of second feature points from an image frame; Matching the plurality of second feature points with a plurality of map points of the map to obtain a plurality of common feature points; and When the number of the plurality of common feature points is not greater than a second threshold, the image frame is determined to be the another key frame.
7. The data processing method according to claim 5, characterized in that: After obtaining the another key frame, the data processing method further comprises: The another key frame is image-recognized by a privacy content recognition module.
8. The data processing method according to claim 3, characterized in that: The plurality of first feature points are not extracted from the at least one image processing area.
9. The data processing method according to claim 1, wherein: Also includes: In a case where the key frame does not include the at least one private content, the map of the environment is updated using the key frame.
10. The data processing method according to claim 1, wherein: The operation of processing the key frame to generate the modified key frame includes: The at least one private content is image-smoothed or image-blurred to generate the at least one image-processed area.
11. An electronic device, characterized in that: Include: an image capture circuit for capturing a plurality of image frames; and A processor is coupled to the image capture circuit and is used to: Setting one of the plurality of image frames as a key frame; In the case where the key frame includes at least one private content, processing the key frame to generate a modified key frame, wherein the modified key frame includes at least one image processing area corresponding to the at least one private content; as well as The modified keyframes are used to update a map of an environment where the electronic device is located.
12. A non-transitory computer-readable storage medium, characterized in that: A computer program is provided for executing a data processing method, wherein the data processing method is applicable to an electronic device in an environment and comprises: Get a key frame; In the case where the key frame includes at least one private content, processing the key frame to generate a modified key frame, wherein the modified key frame includes at least one image processing area corresponding to the at least one private content; as well as A map of the environment is updated using the revised keyframes.