Information processing method and device, equipment and storage medium
By analyzing the number and position of the video data, predicting the line of sight movement trajectory, and masking processing in the overlapping area, the problem of high risk of information leakage in complex environments is solved, and information privacy protection and user experience are improved .
Patent Information
- Application Number
- CN202311039481.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-17
- Publication Date
- 2025-01-14
AI Technical Summary
In complex environments, it is difficult for the prior art to effectively protect the privacy of user information, resulting in a high risk of information leakage, and undifferentiated blocking measures will affect the user experience.
By processing each frame of video data, the number and position of objects are determined, the line of sight movement trajectory is predicted, the overlapping area between the field of view and the information display area is determined based on the preset field of view, and targeted masking is performed in the overlapping area to ensure that The privacy of the information.
It effectively improves the privacy of information, reduces the possibility of information leakage, and ensures user experience.
Smart Images
Figure CN119314074A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the fields of computer technology and financial technology, and in particular to an information processing method, device, equipment, medium and program product. Background Art
[0002] At present, when users are searching for information, cameras are used to collect their facial information and the collected facial information is identified to protect the user's information.
[0003] In the process of realizing the concept disclosed in the present invention, the inventors found that there are at least the following problems in the related technology: when recognizing facial information, if the user is in a complex environment, it is difficult to determine whether there is a risk of information leakage in the current environment and to protect the information in a timely manner, resulting in poor privacy of the information. Summary of the invention
[0004] In view of the above problems, the present disclosure provides an information processing method, apparatus, device, medium and program product.
[0005] According to a first aspect of the present disclosure, there is provided an information processing method, including: processing each frame image of video data, determining the number of objects in each frame image, and obtaining the number of objects corresponding to multiple frames of images, wherein the video data is acquired by an image acquisition module in response to an information viewing request for a first object, and the first object is the object that triggers the information viewing request; in a case where it is determined that there is a risk of displaying information corresponding to the information viewing request according to the number of objects corresponding to the multiple frames of images, determining the position distance between a second object in a target frame image of the video data and an information display area, wherein the second object is an object other than the first object in the target frame image; determining a predicted line of sight movement trajectory of the second object in the video data on the information display area based on the position distance; based on a preset field of view range radius, determining an overlapping area between the field of view area of the second object and the information display area; in a case where the overlapping area includes the predicted line of sight movement trajectory, masking the information displayed in the overlapping area.
[0006] According to an embodiment of the present disclosure, before determining the position distance between the second object in the target frame image of the video data and the information display area, the method further includes:
[0007] Based on the timestamp included in each frame image, a quantity sequence including multiple object quantities is generated; it is detected whether there are multiple continuous numbers of target objects in the quantity sequence, and the multiple continuous numbers of target objects include the number of objects corresponding to the target frame image, and a detection result is generated, wherein the number of target objects is the number of objects greater than a preset number; when the detection result indicates that there are multiple continuous numbers of target objects, and the multiple continuous numbers of target objects include the number of objects corresponding to the target frame image, it is determined that there is a risk in displaying the information corresponding to the information viewing request.
[0008] According to an embodiment of the present disclosure, determining a predicted sight line movement trajectory of a second object in the information display area in the video data based on the position distance includes:
[0009] Facial features are extracted from multiple target frame images in the video data to obtain multiple pupil positions, wherein the target frame image is an image including a second object in the video data; for each pupil position, the predicted sight point coordinates of the second object on the information display area are determined based on the position distance; and according to the timestamp included in the target frame image, the predicted sight motion trajectories corresponding to the multiple predicted sight point coordinates are determined.
[0010] According to an embodiment of the present disclosure, determining the overlap area between the field of view area of the second object and the information display area based on a preset field of view range radius includes:
[0011] Taking the position of the second object as the origin and based on the preset field of view range radius, the field of view area of the second object is determined, wherein the field of view area is a three-dimensional area; when there is a section area between the information display area and the field of view area, the section area is determined as the overlapping area.
[0012] According to an embodiment of the present disclosure, before performing mask processing on the information displayed in the overlapped area, the method further includes:
[0013] Determine whether the predicted sight line movement trajectory is within the overlapping area, or whether the predicted sight line movement trajectory has an intersection with the edge line of the overlapping area, and obtain a determination result; when the predicted sight line movement trajectory is within the overlapping area, or the predicted sight line movement trajectory has an intersection with the edge line of the overlapping area, determine that the determination result represents that the overlapping area includes the predicted sight line movement trajectory.
[0014] According to an embodiment of the present disclosure, before processing each frame of video data to determine the number of objects in each frame and obtaining the number of objects corresponding to multiple frames of images, the method further includes:
[0015] Perform facial feature extraction on the first object in any frame image of the video data to obtain a feature vector; calculate the similarity between the feature vector and a preset feature vector to obtain a feature similarity; when the feature similarity is greater than the preset similarity, determine the number of objects in each frame image to mask the information based on the number of multiple objects; when the feature similarity is less than or equal to the preset similarity, interrupt the visual display of information corresponding to the information viewing request.
[0016] According to an embodiment of the present disclosure, the above method further includes:
[0017] In response to the received information restoration request, the masked information in the overlapping area is restored to a normal display state.
[0018] A second aspect of the present disclosure provides an information processing device, including:
[0019] The quantity determination module is used to process each frame of video data, determine the number of objects in each frame, and obtain the number of objects corresponding to multiple frames of images, wherein the video data is acquired by the image acquisition module in response to an information viewing request for a first object, and the first object is the object that triggers the information viewing request.
[0020] A distance determination module is used to determine the position distance between a second object in a target frame image of video data and an information display area when it is determined that there is a risk of displaying information corresponding to an information viewing request based on the number of objects corresponding to multiple frame images, wherein the second object is an object other than the first object in the target frame image.
[0021] The trajectory prediction module is used to determine the predicted sight line movement trajectory of the second object in the video data on the information display area based on the position distance.
[0022] The overlap determination module is used to determine the overlap area between the field of view area of the second object and the information display area based on a preset field of view range radius.
[0023] The information masking module is used to mask the information displayed in the overlapping area when the overlapping area includes the predicted line of sight movement trajectory.
[0024] A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the above method.
[0025] The fourth aspect of the present disclosure also provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the above method.
[0026] The fifth aspect of the present disclosure also provides a computer program product, including a computer program, which implements the above method when executed by a processor.
[0027] According to the information processing method, device, equipment, medium and program product provided by the present disclosure, by determining the number of objects corresponding to the multiple frames of images in the video data, it is determined whether there is a risk in the environment where the information is currently displayed. In the case of a risk, the line of sight movement trajectory is predicted based on the position distance between the second object in the video data and the information display area. And in the case where the overlapping area includes the predicted line of sight movement trajectory, the information of the overlapping area is masked in a targeted manner. Under the premise of ensuring the user experience, the privacy of the information is effectively improved and the possibility of information leakage is reduced. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:
[0029] Figure 1 The application scenario diagram of the information processing method, apparatus, device, medium and program product according to the embodiments of the present disclosure is schematically shown;
[0030] Figure 2 The flowchart of the information processing method according to the embodiment of the present disclosure is schematically shown;
[0031] Figure 3 A schematic diagram schematically shows a scenario of an information processing method according to an embodiment of the present disclosure;
[0032] Figure 4 The information mask diagram of the information processing method according to the embodiment of the present disclosure is schematically shown;
[0033] Figure 5 A block diagram schematically shows a structure of an information processing device according to an embodiment of the present disclosure; and
[0034] Figure 6 A block diagram of an electronic device suitable for implementing an information processing method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION
[0035] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.
[0036] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.
[0037] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.
[0038] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).
[0039] In the technical solution of the present invention, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of relevant data comply with the relevant laws, regulations and standards of relevant countries and regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.
[0040] Currently, there is a lack of effective privacy protection measures when users query account transaction information. When users view personal account transaction information through mobile devices, their privacy information is easily exposed, which affects the user's sense of security in using functions.
[0041] In the related technology, a camera can be used to collect the user's facial information. The collected facial information is recognized, and if the recognition is successful, the information is displayed normally. If the recognition is unsuccessful, the information is indiscriminately shielded to achieve privacy protection.
[0042] When identifying the user's facial information, it is impossible to identify the scene in which the user is located, thereby providing reasonable privacy protection for the information the user is querying. If the user queries account transaction information in a complex public environment, it is very easy to cause the leakage of sensitive information. Indiscriminately shielding information will affect the user's acquisition of key information, resulting in a poor user experience.
[0043] In view of this, the embodiments of the present disclosure provide an information processing method, device, electronic device, readable storage medium and computer program product, the method comprising: processing each frame image of video data, determining the number of objects in each frame image, and obtaining the number of objects corresponding to multiple frames of images, wherein the video data is acquired by an image acquisition module in response to an information viewing request for a first object, and the first object is the object that triggers the information viewing request; when it is determined that there is a risk of displaying information corresponding to the information viewing request according to the number of objects corresponding to the multiple frames of images, determining the position distance between the second object in the target frame image of the video data and the information display area, wherein the second object is an object other than the first object in the target frame image; determining the predicted line of sight movement trajectory of the second object in the video data on the information display area based on the position distance; determining the overlapping area between the field of view area of the second object and the information display area based on a preset field of view range radius; when the overlapping area includes the predicted line of sight movement trajectory, masking the information displayed in the overlapping area.
[0044] Figure 1 The application scenario diagram of the information processing method, apparatus, device, medium and program product according to the embodiments of the present disclosure is schematically shown.
[0045] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104 and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.
[0046] The user may use at least one of the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications may be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).
[0047] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.
[0048] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.
[0049] It should be noted that the information processing method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the information processing device provided in the embodiment of the present disclosure can generally be set in the server 105. The information processing method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the information processing device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.
[0050] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.
[0051] The following will be based on Figure 1 The scene described by Figure 2 to Figure 4 The information processing method of the disclosed embodiment is described in detail.
[0052] Figure 2 The flowchart of the information processing method according to the embodiment of the present disclosure is schematically shown.
[0053] like Figure 2 As shown, the information processing of this embodiment includes operations S210 to S250.
[0054] In operation S210, each frame of the video data is processed to determine the number of objects in each frame, and the number of objects corresponding to the multiple frames of the image are obtained, wherein the video data is acquired by the image acquisition module in response to the information viewing request for the first object, and the first object is the object that triggers the information viewing request.
[0055] In operation S220, when it is determined that there is a risk in displaying information corresponding to the information viewing request based on the number of objects corresponding to the multiple frame images, the position distance between the second object in the target frame image of the video data and the information display area is determined, wherein the second object is an object other than the first object in the target frame image.
[0056] In operation S230, a predicted sight line movement trajectory of a second object in the information display area in the video data is determined based on the position distance.
[0057] In operation S240, based on a preset field of view radius, an overlap area between the field of view area of the second object and the information display area is determined.
[0058] In operation S250 , when the overlapped area includes the predicted sight line movement trajectory, mask processing is performed on the information displayed in the overlapped area.
[0059] According to an embodiment of the present disclosure, in response to an information viewing request triggered by a first object, the image acquisition module is controlled to capture the first object that triggers the information viewing request to obtain video data. The video data may be video data of a preset duration, such as 3 seconds of video data. The video data includes facial information of the first object and environmental information of the first object. The image acquisition module may be a front-facing camera installed on an electronic device, and the electronic device also includes a display screen, that is, an information display area, for displaying the information requested to be viewed by the object.
[0060] According to an embodiment of the present disclosure, each frame of the video data is processed to determine the number of objects in each frame. Specifically, each frame may include not only the first object, but also other objects in the environment information where the first object is located. Each frame of the image is input into the face information extraction model to determine the number of objects in each frame of the image, and then obtain the number of objects corresponding to the multiple frames of the image.
[0061] According to an embodiment of the present disclosure, it is determined whether there is a risk in displaying information corresponding to an information viewing request based on the number of objects corresponding to multiple frames of images. Specifically, a threshold may be set to determine the number of multiple object numbers that is greater than the threshold, and determine whether there is a risk based on the number greater than the threshold. It is also possible to determine whether there are multiple object numbers that are continuous and greater than the threshold among the multiple object numbers, and then determine whether there is a risk.
[0062] According to an embodiment of the present disclosure, when it is determined that there is a risk in displaying information corresponding to an information viewing request, a second object in a target frame image of the video data is determined. The second object is an object other than the first object in the target frame image, which may be one or more. The target frame image may be an ending frame image in the video data, or an adjacent frame image adjacent to the ending frame. Specifically, the process of determining the second object includes determining facial information of the second object.
[0063] According to an embodiment of the present disclosure, the distance measuring module is controlled to illuminate the second object based on the facial information of the second object, and the position distance between the second object and the information display area is obtained. Specifically, the center of the pupil line of the second object can be illuminated to obtain the position distance. Among them, the distance measuring module is an infrared distance measuring device installed on an electronic device, which can emit infrared rays to the second object and can track the second object. The information display area is a display screen of the electronic device, and the distance measuring module and the information display area are in the same plane.
[0064] According to the embodiment of the present disclosure, feature extraction is performed on the face information of the second object, and the rotation position of the pupil in the eye is determined according to the pupil feature of the second object. The rotation position is combined with the position information to determine the predicted rotation trajectory, and the predicted rotation trajectory is mapped to the information display area to obtain the predicted sight line movement trajectory.
[0065] According to the embodiments of the present disclosure, each object has a corresponding visual field, and the specific content of the information can be clearly seen in the visual field. The visual field can be determined according to a preset visual field radius, wherein the preset visual field radius is determined based on normal vision indicators. Specifically, the center of the pupil line of the second object is taken as the dot, and a circle is drawn based on the preset visual field radius to determine the visual field.
[0066] According to an embodiment of the present disclosure, it is determined whether there is an overlapping area between the information display area and the field of view area. The information display area is a two-dimensional plane area, and the field of view area is a three-dimensional area. Therefore, if there is an overlapping area, the overlapping area is a two-dimensional plane area within the information display area. If there is no overlapping area, it means that the user cannot view the content of the information in the information display area in the field of view, and the information can be displayed normally.
[0067] According to an embodiment of the present disclosure, when it is determined that there is an overlapping area, it is determined whether the overlapping area includes a predicted line of sight motion trajectory mapped in the information display area. If the predicted line of sight motion trajectory is included, it means that the second object is viewing the information in the information display area at the same time as the first object. Therefore, the information in the overlapping area is masked to achieve privacy protection of the information, and at the same time can remind the first object to remain vigilant. Specifically, the masking process can be to encode the information or blur the information so that the second object cannot view it.
[0068] According to the embodiments of the present disclosure, if the overlapped area does not include the predicted sight line movement trajectory, the predicted sight line movement trajectory mapped to the information display area may have errors, and the actual sight line movement trajectory of the second object is not on the plane where the information display area is located. Therefore, it is determined that the second object cannot view the content of the information in the information display area, and the information can be displayed normally.
[0069] According to the embodiments of the present disclosure, by determining the number of objects corresponding to the multiple frames of images in the video data, it is determined whether there is a risk in the environment in which the information is currently displayed. In the case of a risk, the line of sight movement trajectory is predicted based on the position distance between the second object in the video data and the information display area. And in the case where the overlapping area includes the predicted line of sight movement trajectory, the information in the overlapping area is masked in a targeted manner. Under the premise of ensuring the user experience, the privacy of the information is effectively improved and the possibility of information leakage is reduced.
[0070] According to an embodiment of the present disclosure, before determining the position distance between the second object in the target frame image of the video data and the information display area, the following operations may also be included:
[0071] Based on the timestamp included in each frame image, a quantity sequence including multiple object quantities is generated; it is detected whether there are multiple continuous numbers of target objects in the quantity sequence, and the multiple continuous numbers of target objects include the number of objects corresponding to the target frame image, and a detection result is generated, wherein the number of target objects is the number of objects greater than a preset number; when the detection result indicates that there are multiple continuous numbers of target objects, and the multiple continuous numbers of target objects include the number of objects corresponding to the target frame image, it is determined that there is a risk in displaying the information corresponding to the information viewing request.
[0072] According to the embodiments of the present disclosure, each frame of the video data has a timestamp. After determining the number of objects in each frame, the numbers of objects corresponding to the multiple frames are sorted according to the timestamp of each frame to obtain a quantity sequence. Specifically, in the sorting process, the timestamps can be sorted from near to far from the current moment, or from far to near from the current moment.
[0073] According to an embodiment of the present disclosure, it is detected whether there are multiple continuous numbers of target objects in the number sequence, and further detected whether the multiple continuous numbers of target objects include the number of objects corresponding to the target frame image, and a detection result is generated. The target frame image can be the end frame image in the video data, or it can be an adjacent frame image adjacent to the end frame.
[0074] According to an embodiment of the present disclosure, if there are multiple continuous numbers of target objects in the detection number sequence, and the multiple continuous numbers of target objects include the number of objects corresponding to the target frame image, it means that the second object continues to appear in the video data, and the second object still exists at the end of the video data. Therefore, the second object may continuously view the information in the information display area in a static manner, indicating that there is a risk in displaying the information corresponding to the information viewing request.
[0075] According to an embodiment of the present disclosure, if there are multiple continuous numbers of target objects in the detection number sequence, but the multiple continuous numbers of target objects do not include the number of objects corresponding to the target frame image, it means that the second object continues to appear in the video data, but the second object does not exist at the end of the video data. It is possible that there is a dynamic second object in the environment where the first object is located, and at the end of the video data, the second object has disappeared in the environment where the first object is located. Therefore, the second object may have just passed through the environment where the first object is located, and did not continuously view the information in the information display area, so it is determined that there is no risk in displaying the information corresponding to the information viewing request.
[0076] According to the embodiments of the present disclosure, if there are no multiple continuous numbers of target objects in the detection number sequence, it means that the second object is dynamic in the environment where the first object is located, and does not continuously view the information in the information display area, so it is determined that there is no risk in displaying the information corresponding to the information viewing request. Determining whether there is a risk in displaying the information corresponding to the information viewing request based on the number of objects corresponding to the multiple frames of images makes the risk determination more accurate, and takes into account whether the second object is continuously viewing the information in the information display area, so as to take subsequent measures, further improving the privacy of the information.
[0077] According to an embodiment of the present disclosure, determining a predicted sight line movement trajectory of a second object in the information display area in the video data based on the position distance may include the following operations:
[0078] Facial features are extracted from multiple target frame images in the video data to obtain multiple pupil positions, wherein the target frame image is an image including a second object in the video data; for each pupil position, the predicted sight point coordinates of the second object on the information display area are determined based on the position distance; and according to the timestamp included in the target frame image, the predicted sight motion trajectories corresponding to the multiple predicted sight point coordinates are determined.
[0079] According to an embodiment of the present disclosure, a face information extraction model is used to extract face features from a target frame image, and the rotation position of the pupil of the second object in each target frame image, that is, the current pupil position, is obtained. Taking the pupil position as the starting point, a trajectory point is generated in a direction perpendicular to the information display area based on the position distance. The trajectory point of each target frame image is determined, and multiple trajectory points are sequentially connected according to the timestamp in each target frame image to obtain a predicted rotation trajectory.
[0080] According to the embodiment of the present disclosure, a rectangular coordinate system is established with the lower left corner of the information display area as the origin, so that the information display area is in the first quadrant of the rectangular coordinate system. Each trajectory point is mapped to the information display area to obtain the predicted line of sight movement trajectory φ=([x1,y1],[x2,y2],...,[x n ,y n ]). Wherein, x represents the horizontal coordinate of the position of the trajectory point of the second object in the information display area, and y represents the vertical coordinate of the position of the trajectory point of the second object in the information display area. Determining the predicted sight line movement trajectory by the above method makes the determination more accurate and reasonable.
[0081] Figure 3 A scenario diagram of an information processing method according to an embodiment of the present disclosure is schematically shown.
[0082] According to an embodiment of the present disclosure, determining the overlap area between the field of view area of the second object and the information display area based on a preset field of view range radius may include the following operations:
[0083] Taking the position of the second object as the origin and based on the preset field of view range radius, the field of view area of the second object is determined, wherein the field of view area is a three-dimensional area; when there is a section area between the information display area and the field of view area, the section area is determined as the overlapping area.
[0084] According to the embodiments of the present disclosure, Figure 3 As shown, the information display area 301 includes an image acquisition module 302 and a distance measurement module 303. The image acquisition module 301 acquires video data of a first object 304 and a second object 305. The distance measurement module 303 illuminates the second object 305 to obtain a position distance for determining a predicted sight line motion trajectory 306.
[0085] According to an embodiment of the present disclosure, the center of the pupil line of the second object 305 is taken as the origin, and the field of view area 307 of the second object 305 is determined based on the preset field of view range radius. The field of view area 307 and the section area of the information display area 301 are determined as the overlap area 308, so that it is more accurate to determine that the second object 305 views part of the information in the information display area 301.
[0086] Figure 4 The information mask diagram of the information processing method according to the embodiment of the present disclosure is schematically shown.
[0087] According to an embodiment of the present disclosure, before performing mask processing on the information displayed in the overlapped area, the following operations may also be included:
[0088] Determine whether the predicted sight line movement trajectory is within the overlapping area, or whether the predicted sight line movement trajectory has an intersection with the edge line of the overlapping area, and obtain a determination result; when the predicted sight line movement trajectory is within the overlapping area, or the predicted sight line movement trajectory has an intersection with the edge line of the overlapping area, determine that the determination result represents that the overlapping area includes the predicted sight line movement trajectory.
[0089] According to the embodiments of the present disclosure, it is possible to directly determine whether the predicted sight line movement trajectory is within the overlapped area, specifically including whether the predicted sight line movement trajectory is partially within the overlapped area or completely within the overlapped area, thereby obtaining a confirmed result. Alternatively, it is determined whether the predicted sight line movement trajectory has an intersection with the edge line of the overlapped area. If there is an intersection, it means that the predicted sight line movement trajectory is partially within the overlapped area. If there is no intersection, the relative position of the predicted sight line movement trajectory and the overlapped area is further determined, thereby obtaining a confirmed result. Figure 4 As shown, it is determined that the predicted sight line movement trajectory is within the overlapped area, and the information in the overlapped area is masked. By determining whether the overlapped area contains the predicted sight line movement trajectory by the above method, various position situations between the predicted sight line movement trajectory and the overlapped area can be considered, making the determination result more accurate.
[0090] According to an embodiment of the present disclosure, before processing each frame of video data, determining the number of objects in each frame, and obtaining the number of objects corresponding to multiple frames of images, the following operations may also be included:
[0091] Perform facial feature extraction on the first object in any frame image of the video data to obtain a feature vector; calculate the similarity between the feature vector and a preset feature vector to obtain a feature similarity; when the feature similarity is greater than the preset similarity, determine the number of objects in each frame image to mask the information based on the number of multiple objects; when the feature similarity is less than or equal to the preset similarity, interrupt the visual display of information corresponding to the information viewing request.
[0092] According to an embodiment of the present disclosure, an image of a first object in any frame of video data is input into a face feature extraction model, and the face features of the first object are extracted to obtain a feature vector λ=(λ1,λ2,...,λ n ). The feature vector is similar to the preset feature vector λ'=(λ'1,λ'2,...,λ'3) to obtain the feature similarity. The calculation formula is as follows:
[0093] d(λ,λ')=|λ1-λ'1|+|λ2-λ'2|+...+|λ n -λ' n | (1)
[0094] Among them, d(λ,λ') represents the feature similarity.
[0095] According to the embodiments of the present disclosure, when the feature similarity is greater than the preset similarity, it indicates that the first object is a preset object, and further risk detection can be performed. When the feature similarity is less than or equal to the preset similarity, it indicates that the first object is not a preset object, and the execution of the target application is immediately terminated to interrupt the visual display of the information corresponding to the information viewing request, thereby minimizing the risk of information leakage.
[0096] According to an embodiment of the present disclosure, the information protection method may further include the following operations:
[0097] In response to the received information restoration request, the masked information in the overlapping area is restored to a normal display state.
[0098] According to an embodiment of the present disclosure, the first object can trigger an information restoration request to restore the masked information in the overlapping area to a normal display state. When an information viewing request is received next time, risk detection is performed again to determine whether to mask the information in the information display area, so as to effectively improve the user experience while ensuring privacy.
[0099] Based on the above information processing method, the present disclosure also provides an information processing device. Figure 5 The device is described in detail.
[0100] Figure 5 The structure block diagram of the information processing device according to the embodiment of the present disclosure is schematically shown.
[0101] like Figure 5 As shown, the information processing device 500 of this embodiment includes a quantity determination module 510 , a distance determination module 520 , a trajectory prediction module 530 , an overlap determination module 540 and an information mask module 550 .
[0102] The quantity determination module 510 is used to process each frame of the video data, determine the number of objects in each frame, and obtain the number of objects corresponding to the multiple frames of the image, wherein the video data is acquired by the image acquisition module in response to the information viewing request for the first object, and the first object is the object that triggers the information viewing request. In one embodiment, the quantity determination module 510 can be used to perform the operation S210 described above, which will not be repeated here.
[0103] The distance determination module 520 is used to determine the position distance between the second object in the target frame image of the video data and the information display area when it is determined that there is a risk of displaying the information corresponding to the information viewing request according to the number of objects corresponding to the multiple frame images, wherein the second object is an object other than the first object in the target frame image. In one embodiment, the distance determination module 520 can be used to perform the operation S220 described above, which will not be repeated here.
[0104] The trajectory prediction module 530 is used to determine the predicted sight line movement trajectory of the second object in the video data on the information display area based on the position distance. In one embodiment, the trajectory prediction module 530 can be used to perform the operation S230 described above, which will not be repeated here.
[0105] The overlap determination module 540 is used to determine the overlap area between the field of view area of the second object and the information display area based on a preset field of view range radius. In one embodiment, the overlap determination module 540 can be used to perform the operation S240 described above, which will not be repeated here.
[0106] The information mask module 550 is used to mask the information displayed in the overlapped area when the overlapped area includes the predicted sightline movement trajectory. In one embodiment, the information mask module 550 can be used to perform the operation S250 described above, which will not be described in detail here.
[0107] According to the embodiments of the present disclosure, by determining the number of objects corresponding to the multiple frames of images in the video data, it is determined whether there is a risk in the environment in which the information is currently displayed. In the case of a risk, the line of sight movement trajectory is predicted based on the position distance between the second object in the video data and the information display area. And in the case where the overlapping area includes the predicted line of sight movement trajectory, the information in the overlapping area is masked in a targeted manner. Under the premise of ensuring the user experience, the privacy of the information is effectively improved and the possibility of information leakage is reduced.
[0108] According to an embodiment of the present disclosure, the distance determination module 520 includes a sequence generation submodule, a result generation submodule, and a risk determination submodule.
[0109] The sequence generation submodule is used to generate a quantity sequence including a plurality of object quantities according to a timestamp included in each frame of the image.
[0110] The result generation submodule is used to detect whether there are multiple continuous numbers of target objects in the number sequence, and the multiple continuous numbers of target objects include the number of objects corresponding to the target frame image, and generate a detection result, wherein the number of target objects is the number of objects greater than a preset number.
[0111] The risk determination submodule is used to determine that there is a risk in displaying information corresponding to the information viewing request when the detection result indicates that there are multiple continuous target object numbers, and the multiple continuous target object numbers include the number of objects corresponding to the target frame image.
[0112] According to an embodiment of the present disclosure, the trajectory prediction module 530 includes a feature extraction submodule, a landing point determination submodule, and a trajectory determination submodule.
[0113] The feature extraction submodule is used to extract facial features from multiple target frame images in the video data to obtain multiple pupil positions, wherein the target frame image is an image including the second object in the video data.
[0114] The landing point determination submodule is used to determine the predicted sight point coordinates of the second object on the information display area based on the position distance for each pupil position.
[0115] The trajectory determination submodule is used to determine the predicted sight line movement trajectory corresponding to the multiple predicted sight line landing point coordinates according to the timestamp included in the target frame image.
[0116] According to an embodiment of the present disclosure, the coincidence determination module 540 includes a field of view determination submodule and a coincidence determination submodule.
[0117] The field of view determination submodule is used to determine the field of view area of the second object based on a preset field of view range radius with the position of the second object as the origin, wherein the field of view area is a three-dimensional area.
[0118] The overlap determination submodule is used to determine the section area as the overlap area when there is a section area between the information display area and the visual field area.
[0119] According to an embodiment of the present disclosure, the information processing device 500 further includes a result determination module and an area determination module.
[0120] The result determination module is used to determine whether the predicted sight line movement trajectory is within the overlapped area, or whether the predicted sight line movement trajectory has an intersection with the edge line of the overlapped area, to obtain a determination result.
[0121] The area determination module is used to determine that the determination result represents that the overlapping area includes the predicted sight line movement trajectory when the predicted sight line movement trajectory is within the overlapping area or the predicted sight line movement trajectory has an intersection with the edge line of the overlapping area.
[0122] According to an embodiment of the present disclosure, the information processing device 500 further includes a feature extraction module, a vector calculation module, a second quantity determination module, and a display interruption module.
[0123] The feature extraction module is used to extract the facial features of the first object in any frame image of the video data to obtain a feature vector.
[0124] The vector calculation module is used to calculate the similarity between the feature vector and the preset feature vector to obtain feature similarity.
[0125] The second quantity determination module is used to determine the number of objects in each frame of the image when the feature similarity is greater than a preset similarity, so as to perform mask processing on the information based on multiple object quantities.
[0126] The display interruption module is used to interrupt the visual display of information corresponding to the information viewing request when the feature similarity is less than or equal to the preset similarity.
[0127] According to an embodiment of the present disclosure, the information processing device 500 further includes an information recovery module.
[0128] The information recovery module is used to restore the masked information in the overlapped area to a normal display state in response to a received information restoration request.
[0129] According to an embodiment of the present disclosure, any multiple modules of the quantity determination module 510, the distance determination module 520, the trajectory prediction module 530, the overlap determination module 540 and the information mask module 550 can be combined in one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the quantity determination module 510, the distance determination module 520, the trajectory prediction module 530, the overlap determination module 540 and the information mask module 550 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in any appropriate combination of any of them. Alternatively, at least one of the quantity determination module 510 , the distance determination module 520 , the trajectory prediction module 530 , the overlap determination module 540 , and the information mask module 550 may be at least partially implemented as a computer program module, which may perform corresponding functions when executed.
[0130] Figure 6 A block diagram of an electronic device suitable for implementing an information processing method according to an embodiment of the present disclosure is schematically shown.
[0131] like Figure 6 As shown, the electronic device 600 according to an embodiment of the present disclosure includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 into a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include an onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.
[0132] In RAM 603, various programs and data required for the operation of electronic device 600 are stored. Processor 601, ROM 602 and RAM 603 are connected to each other via bus 604. Processor 601 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 602 and / or RAM 603. It should be noted that the program can also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.
[0133] According to an embodiment of the present disclosure, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to the bus 604. The electronic device 600 may further include one or more of the following components connected to the input / output (I / O) interface 605: an input portion 606 including a keyboard, a mouse, etc.; an output portion 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 608 including a hard disk, etc.; and a communication portion 609 including a network interface card such as a LAN card, a modem, etc. The communication portion 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the input / output (I / O) interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage portion 608 as needed.
[0134] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.
[0135] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present disclosure, a computer-readable storage medium may include the ROM 602 and / or RAM 603 described above and / or one or more memories other than ROM 602 and RAM 603.
[0136] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the information processing method provided by the embodiment of the present disclosure.
[0137] The above functions defined in the system / device of the embodiment of the present disclosure are performed when the computer program is executed by the processor 601. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.
[0138] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 609, and / or installed from a removable medium 611. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.
[0139] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the processor 601, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, apparatus, module, unit, etc. described above can be implemented by a computer program module.
[0140] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).
[0141] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0142] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations and / or combinations are not explicitly described in the present disclosure. In particular, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways without departing from the spirit and teachings of the present disclosure. All of these combinations and / or combinations fall within the scope of the present disclosure.
[0143] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. The scope of the present disclosure is defined by the attached claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.
Claims
1. An information processing method, comprising: Processing each frame of the video data, determining the number of objects in each frame of the image, and obtaining the number of objects corresponding to the multiple frames of the image, wherein the video data is acquired by the image acquisition module in response to the information viewing request for the first object, and the first object is the object that triggers the information viewing request; In a case where it is determined that there is a risk in displaying the information corresponding to the information viewing request according to the number of objects corresponding to the multiple frames of the image, determining a position distance between a second object in the target frame image of the video data and the information display area, wherein the second object is an object other than the first object in the target frame image; Determine, based on the position distance, a predicted sight line movement trajectory of the second object in the video data on the information display area; Based on a preset field of view radius, determining an overlapping area between a field of view area of the second object and the information display area; In the case where the overlapped area includes the predicted line of sight movement trajectory, mask processing is performed on the information displayed in the overlapped area.
2. The method according to claim 1, wherein: Before determining the position distance between the second object in the target frame image of the video data and the information display area, the method further includes: Generate a quantity sequence including a plurality of quantities of the objects according to a timestamp included in each frame of the image; Detecting whether there are multiple continuous numbers of target objects in the number sequence, and the multiple continuous numbers of target objects include the number of objects corresponding to the target frame image, and generating a detection result, wherein the number of target objects is a number of objects greater than a preset number; When the detection result indicates that there are multiple continuous numbers of target objects, and the multiple continuous numbers of target objects include the number of objects corresponding to the target frame image, it is determined that there is a risk in displaying the information corresponding to the information viewing request.
3. The method according to claim 1, wherein: The determining, based on the position distance, a predicted sight line movement trajectory of the second object in the video data on the information display area includes: Extracting facial features from a plurality of target frame images in the video data to obtain a plurality of pupil positions, wherein the target frame images are images in the video data that include the second object; For each pupil position, determining the predicted sight point coordinates of the second object on the information display area based on the position distance; The predicted sight line movement trajectory corresponding to the plurality of predicted sight line landing point coordinates is determined according to the timestamp included in the target frame image.
4. The method according to claim 1, wherein: The determining, based on a preset field of view radius, an overlapping area between the field of view area of the second object and the information display area includes: Taking the position of the second object as the origin and based on the preset field of view radius, determining the field of view area of the second object, wherein the field of view area is a three-dimensional area; In the case where there is a section area between the information display area and the visual field area, the section area is determined as the overlapping area.
5. The method according to claim 1, wherein: Before performing masking on the information displayed in the overlapped area, the method further includes: Determine whether the predicted sight line movement trajectory is within the overlapped area, or whether the predicted sight line movement trajectory has an intersection with an edge line of the overlapped area, and obtain a determination result; When the predicted sight line movement trajectory is within the overlapped area, or when the predicted sight line movement trajectory intersects an edge line of the overlapped area, it is determined that the determination result represents that the overlapped area includes the predicted sight line movement trajectory.
6. The method according to claim 1, wherein: Before processing each frame of the video data to determine the number of objects in each frame of the image and obtaining the number of objects corresponding to the multiple frames of the image, the method further includes: Extracting facial features of the first object in any frame of the video data to obtain a feature vector; Calculate the similarity between the feature vector and a preset feature vector to obtain feature similarity; When the feature similarity is greater than a preset similarity, determining the number of objects in each frame of the image, so as to perform mask processing on the information based on a plurality of the number of objects; When the feature similarity is less than or equal to the preset similarity, the visual display of the information corresponding to the information viewing request is interrupted.
7. The method according to claim 1, further comprising: In response to the received information restoration request, the masked information in the overlapped area is restored to a normal display state.
8. An information protection device, comprising: a quantity determination module, configured to process each frame of the video data, determine the number of objects in each frame of the image, and obtain the number of objects corresponding to the plurality of frames of the image, wherein the video data is acquired by the image acquisition module in response to the information viewing request for the first object, and the first object is the object that triggers the information viewing request; a distance determination module, configured to determine a position distance between a second object in a target frame image of the video data and an information display area when it is determined that there is a risk of displaying the information corresponding to the information viewing request according to the number of objects corresponding to the plurality of frames of the image, wherein the second object is an object other than the first object in the target frame image; a trajectory prediction module, configured to determine a predicted sight line movement trajectory of the second object in the video data on the information display area based on the position distance; An overlap determination module, configured to determine an overlap area between a field of view area of the second object and the information display area based on a preset field of view range radius; An information masking module is used to mask the information displayed in the overlapping area when the overlapping area includes the predicted line of sight movement trajectory.
9. An electronic device, comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors are enabled to execute the method according to any one of claims 1 to 7.
10. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the method according to any one of claims 1 to 7.