Data processing method and apparatus, device, and storage medium

By capturing image data and asking questions in terminal devices, judging and processing potential privacy information, the problem of privacy leakage in multimodal visual language questions and answers is solved, and the security Q&A services and privacy protection for visually impaired people are realized.

WO2024227414A1PCT designated stage expired Publication Date: 2024-11-07BEIJING YOUZHUJU NETWORK TECH CO LTD

Patent Information

Application Number
PCT/CN2024/089622
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-04
Filing Date
2024-04-24
Publication Date
2024-11-07

AI Technical Summary

Technical Problem

In multimodal visual language Q&A applications, the image data entered by the user may contain potential privacy information, resulting in unconscious privacy leakage risks, especially for visually impaired people, which are difficult to detect and protect privacy information.

Method used

By capturing image data and asking questions in the terminal device, determining whether the image data contains potential privacy information, and determining whether the question is an intentional question for these information, performing distinctive privacy protection processing based on the judgment results, such as privacy protection of image data, based on the judgment results, Operation or protection is not performed.

Benefits of technology

Reduces the risk of unconscious privacy leakage of users, improves the security of Q&A based on visual information, and provides visually impaired people with safe and reliable multimodal visual language Q&A services to ensure that potential private information can be answered instantly when asked intentionally. .

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024089622_07112024_PF_FP_ABST
    Figure CN2024089622_07112024_PF_FP_ABST
Patent Text Reader

Abstract

A data processing method and apparatus, a device, and a storage medium. The method comprises: using a device of a user to capture image data and indicate a question for the image data; determining whether the image data contains potential privacy information; if it is determined that the image data contains potential privacy information, determining whether the question is an intentional question for the potential privacy information; and executing privacy protection processing for the image data on the basis of the determination of whether the question is the intentional question for the potential privacy information. Therefore, differentiated privacy protection processing is executed on the image data on the basis of the determination of whether the question is the intentional privacy question, which helps to reduce the risk of unintentional privacy leakage by users and provide the users with instant question-and-answer services in the case of intentional questions.
Need to check novelty before this filing date? Find Prior Art

Description

Data processing method, device, equipment and storage medium

[0001] This application claims priority to the Chinese invention patent application entitled “Method, device, equipment and storage medium for data processing” and application number 202310491084.2, filed on May 4, 2023. The entire contents of that application are incorporated by reference into this application. Technical Field

[0002] Example embodiments of the present disclosure generally relate to the field of computers, and more particularly, to methods, devices, apparatuses, and computer-readable storage media for data processing. Background Art

[0003] With the rapid development of information technology, more and more applications are providing question-and-answer functions, which has brought many conveniences to the majority of users. Applications with question-and-answer functions can output corresponding answers based on the voice or text input by the user. Applications with multimodal visual question answering (VQA) functions can also output audio answers to the image based on voice questions based on the image input by the user. However, it is also expected that the input image can be privacy-protected while achieving multimodal visual question and answer.

[0004] Summary of the Invention

[0005] In a first aspect of the present disclosure, a data processing method is provided. The method includes: capturing image data using a user's device and indicating a question regarding the image data; determining whether the image data contains potentially private information; if it is determined that the image data contains potentially private information, determining whether the question is an intentional question regarding the potentially private information; and performing privacy protection processing on the image data based on the determination of whether the question is an intentional question regarding the potentially private information.

[0006] In a second aspect of the present disclosure, a device for data processing is provided. The device includes: a data capture module configured to capture image data using a user's device and instruct a question regarding the image data; an information determination module configured to determine whether the image data contains potentially private information; a question determination module configured to, if it is determined that the image data contains potentially private information, determine whether the question is an intentional question regarding the potentially private information; and a privacy protection processing module configured to perform privacy protection processing on the image data based on the determination of whether the question is an intentional question regarding the potentially private information.

[0007] In a third aspect of the present disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the electronic device to perform the method of the first aspect.

[0008] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the computer-readable storage medium, and the computer program can be executed by a processor to implement the method of the first aspect.

[0009] It should be understood that the content described in this section is not intended to limit the key features or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0011] FIG1 shows a schematic diagram of an example environment in which embodiments of the present disclosure can be implemented;

[0012] FIG2 shows a flow chart of a data processing process according to some embodiments of the present disclosure;

[0013] FIG3 shows a schematic diagram of a question-and-answer interface according to some embodiments of the present disclosure;

[0014] FIG4 is a schematic diagram showing a data processing flow according to some embodiments of the present disclosure;

[0015] FIG5 shows a schematic structural block diagram of an apparatus for data processing according to some embodiments of the present disclosure; and

[0016] FIG6 illustrates a block diagram of an electronic device in which one or more embodiments of the present disclosure may be implemented. DETAILED DESCRIPTION

[0017] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0018] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below. As used herein, the term "model" can represent the association relationship between various data. For example, the above-mentioned association relationship can be obtained based on a variety of technical solutions currently known and / or to be developed in the future.

[0019] Herein, unless explicitly stated otherwise, executing a step “in response to A” does not mean executing the step immediately after “A” but may include one or more intermediate steps.

[0020] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) must comply with the requirements of relevant laws, regulations and relevant provisions.

[0021] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0022] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the operation requested will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operations of the disclosed technical solution based on the prompt message.

[0023] As an optional but non-limiting embodiment, in response to receiving a user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. Furthermore, the pop-up window may also include a target exploration control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0024] It is understandable that the above notification and user authorization process are merely illustrative and do not limit the embodiments of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the embodiments of the present disclosure.

[0025] As used herein, the term "model" can learn the association between corresponding inputs and outputs from training data, so that after training is completed, corresponding outputs can be generated for given inputs. The generation of the model can be based on machine learning technology. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs by using multiple layers of processing units. A neural network model is an example of a model based on deep learning. In this article, "model" may also be referred to as "machine learning model", "learning model", "machine learning network" or "learning network", and these terms are used interchangeably in this article.

[0026] A "neural network" is a machine learning network based on deep learning. A neural network is capable of processing inputs and providing corresponding outputs. It typically includes an input layer, an output layer, and one or more hidden layers between the input and output layers. Neural networks used in deep learning applications typically include many hidden layers, thereby increasing the depth of the network. The layers of a neural network are connected in sequence so that the output of the previous layer is provided as input to the next layer, where the input layer receives the input of the neural network and the output of the output layer serves as the final output of the neural network. Each layer of a neural network includes one or more nodes (also called processing nodes or neurons), each of which processes the input from the previous layer.

[0027] Generally speaking, machine learning can be roughly divided into three stages, namely the training stage, the testing stage, and the application stage (also known as the inference stage). In the training stage, a given model can be trained using a large amount of training data, and the parameter values ​​are continuously updated iteratively until the model can obtain consistent inferences that meet the expected goals from the training data. Through training, the model can be considered to be able to learn the association between input and output (also known as input-to-output mapping) from the training data. The parameter values ​​of the trained model are determined. In the testing stage, the test input is applied to the trained model to test whether the model can provide the correct output, thereby determining the performance of the model. In the application stage, the model can be used to process the actual input based on the parameter values ​​obtained through training to determine the corresponding output.

[0028] As briefly mentioned above, an application with a multimodal visual language question-answering function can play an audio answer to an image input by a user based on a voice question input by the user.

[0029] Visual-language question answering (VQA) is a multimodal understanding task that requires answering verbal questions based on understanding visual content. Traditionally, applications with multimodal VQA capabilities have leveraged trained question-answering models to implement VQA.

[0030] However, people may sometimes find it difficult to detect whether the image data they capture contains private information that they do not want to disclose, which makes the image data input by users in the question-and-answer scenario based on visual information lead to the risk of unconscious privacy leakage. Specifically, the process of determining the answer by the question-and-answer model may be performed remotely (such as the cloud). Moreover, the question-and-answer model needs to collect a large amount of image data when updating or improving the model, and the collected image data will be provided to the annotation personnel to obtain the annotation information. This will lead to the leakage of private information that the user unconsciously captured in the image data. In particular, for some groups of people, such as the visually impaired, the demand for visual language question-and-answer is greater, and these people often find it more difficult to detect whether the captured image data contains private information that they do not want to disclose due to their limited vision. Therefore, it is expected that in the question-and-answer scenario based on visual data, while providing question-and-answer services, better privacy protection can be achieved.

[0031] The embodiments of the present disclosure propose an improved scheme for data processing. The scheme involves question-answering based on image data (visual data). According to the scheme, image data is collected and questions about the image data are indicated. When it is determined that the image data contains potential privacy information, it is judged whether the question is an intentional question about the potential privacy information. Based on the judgment result, privacy protection processing is performed on the image data. In this way, discriminative privacy protection processing can be performed on the image data based on the judgment of whether it is an intentional privacy question. On the one hand, this helps to reduce the risk of unconscious privacy leakage of users and improve security in the context of question-answering based on visual information. On the other hand, it can help provide users with instant question-answering services in the case of intentional questions about potential privacy information.

[0032] Furthermore, the question-and-answer solution proposed in this disclosure can effectively assist users, especially those with persistent or temporary vision impairment or obstruction, by safely and reliably implementing multimodal visual language question-and-answering. It should be understood that the solution provided by the embodiments of this disclosure can provide convenience for specific groups of people, but this does not imply any discrimination against specific groups of people.

[0033] FIG1 illustrates a schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented. In this example environment 100, an application 120 is installed on a terminal device 110. A user 140 can interact with the application 120 via the terminal device 110 and / or its attached devices. The application 120 is an application that has at least a visual language question-answering function.

[0034] In some embodiments, the terminal device 110 communicates with the computing device 132 in the cloud computing environment 130 to provide services for the application 120. The terminal device 110 can be any type of mobile terminal, fixed terminal, or portable terminal, including a mobile phone, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a media computer, a multimedia tablet, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio receiver, an e-book device, a gaming device, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some embodiments, the terminal device 110 can also support any type of interface for the user (such as "wearable" circuitry, etc.).

[0035] The terminal device 110 may, for example, include a sensor of an appropriate type for detecting user gestures. For example, the terminal device 110 may include a touch screen for detecting various types of gestures made by the user on the touch screen. Alternatively or additionally, the terminal device 110 may also include other appropriate types of sensing devices such as proximity sensors to detect various types of gestures made by the user within a predetermined distance above the screen. The terminal device 110 may also include, for example, a sound collection device (such as a microphone) for collecting user audio, a sound playback device (such as a speaker) for playing audio, an image collection device (such as a camera, a webcam, etc.) for collecting images, and a display device (such as a display screen, which may be a touch screen) for displaying an interface, etc.

[0036] Computing device 132 in cloud computing environment 130 is a remote device relative to terminal device 110. Computing device 132 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks, and big data and artificial intelligence platforms. Computing device 132 can include, for example, computing systems / servers such as mainframes, edge computing nodes, computing devices in a cloud environment, and so on. Computing device 132 provides backend services for application 120 in terminal device 110.

[0037] In some embodiments discussed below, visual language question-answering functionality can be implemented using multiple models with various functions. One or more of these models can be remotely deployed in the computing device 132, and the terminal device 110 can utilize these multiple models to implement corresponding functions through communication with the computing device 132 in the cloud computing environment 130. This can conserve resources and power on the terminal device 110 and utilize the powerful resources of the server to improve computing efficiency. In some embodiments, one or more of these models can also be deployed locally on the terminal device 110. This can be selected based on actual circumstances.

[0038] In some embodiments, in the environment 100 of FIG. 1 , if the application 120 is active, the terminal device 110 may present an interface 150 of the application 120. Through the interface 150, the application 120 may provide the user 140 with one or more services related to the question-and-answer function, including voice capture, image capture, voice playback, text display, and the like.

[0039] It should be understood that the structure and functionality of environment 100 are described for exemplary purposes only and do not imply any limitation on the scope of the present disclosure.

[0040] Some example embodiments of the present disclosure will be described below with continued reference to the accompanying drawings.

[0041] 2 shows a flow chart of a process 200 for data processing according to some embodiments of the present disclosure. The process 200 may be implemented at the terminal device 110. For ease of discussion, the process 200 will be described with reference to the environment 100 of FIG. 1 .

[0042] At block 210 , the terminal device 110 captures image data and indicates a question regarding the image data.

[0043] In some embodiments, the terminal device 110 may capture image data and indicate questions about the image data in response to detecting a question-and-answer initiation operation. Specifically, the terminal device 110 directly detects the question-and-answer initiation operation initiated by the user. For example, the terminal device 110 may determine that a question-and-answer initiation operation is detected in response to detecting a question-and-answer initiation voice (e.g., "turn on the question-and-answer function"). For another example, the terminal device 110 may determine that a question-and-answer initiation operation is detected in response to detecting a preset operation on a hardware button (e.g., a press operation, a long press operation, etc.). In some embodiments, after detecting the question-and-answer initiation operation, the terminal device 110 runs an application 120 with a question-and-answer function, captures image data, and indicates questions about the image data.

[0044] The terminal device 110 may capture image data using an image acquisition device. The image data may be in any form (e.g., a still image or a video clip), at any resolution, and in any format (e.g., PNG, JPG, etc.). Alternatively or additionally, the image data may be data pre-stored in the terminal device 110.

[0045] Questions can include multiple forms, that is, the terminal device 110 can capture questions in multiple forms. Questions can include, for example, questions captured in text form, and the terminal device 110 can directly obtain the text sequence of questions input by the user (for example, obtaining the text sequence of "How many cups are there in the image?"). In order to ensure the convenience and ease of operation of questions and answers, questions can also include questions captured in voice form, and the terminal device 110 can capture the voice data of the user's questions through a sound collection device. The voice data can be in any language (for example, Chinese, English, Japanese, etc.), any time length (for example, 3s, 5s, etc.) and any timbre. It can be understood that questions can also be questions captured in any other appropriate form.

[0046] In the case where the question is captured in the form of voice, in some embodiments, after the terminal device 110 launches the application 120, a recording interface including at least a recording control may be presented via a display device. In response to detecting a predetermined operation on the recording control, the terminal device 110 may determine that a question-and-answer initiation operation has been detected, and then capture image data and voice data. The predetermined operation on the recording control may include, for example, a click operation, a slide operation, a long press operation, etc., without limitation herein. In some embodiments, the predetermined operation on the recording control may also be initiated via voice or other instructions.

[0047] In some embodiments, during the process of capturing image data and voice data, the terminal device 110 may also stop capturing image data and voice data in response to receiving a capture end operation. Specifically, the terminal device 110 may determine that a capture end operation has been detected in response to detecting a voice such as "stop collecting data". The terminal device 110 may also determine that a capture end operation has been detected in response to detecting a preset operation on a hardware button (such as a press operation, a long press operation, etc.). The terminal device 110 may also determine that a capture end operation has been detected in response to detecting another predetermined operation on a recording control in the recording interface (such as a click operation, a release press operation, etc.).

[0048] Referring to Figure 3, a schematic diagram of a recording interface 300 according to some embodiments of the present disclosure is shown. Recording interface 300 may include a control display area 330, which displays at least a recording control 332. Terminal device 110 may determine that a Q&A initiation operation has been detected in response to detecting a predetermined operation on recording control 332. Terminal device 110 may then display text such as "Recording" in text prompt area 310 to notify the user that terminal device 110 is currently capturing data.

[0049] In some embodiments, in response to receiving a predetermined operation, the terminal device 110 changes the presentation of the recording control 332 (e.g., by changing the color or size of the recording control 332) to indicate that the terminal device 110 is capturing voice data and image data. Accordingly, in response to receiving a capture end operation, the terminal device 110 can stop capturing voice data and image data. In response to receiving a capture end operation, the terminal device 110 can switch the presentation of the recording control 332 back to the state before capture.

[0050] In some embodiments, the terminal device 110 can convert the captured voice data into text and display it in the text prompt area 310. As shown in Figure 3, after the terminal device 110 captures the voice data, it displays the text corresponding to the voice data, "What is the card number?", in the text prompt area 310. Simultaneously, the terminal device 110 can present the image data currently being captured by the terminal device 110 in the image display area 320. The voice-to-text conversion can be achieved using speech-to-text technology, which can be performed locally on the terminal device 110 or on a remote server.

[0051] In some embodiments, to ensure the accuracy of the question indicated by the determined voice data, the terminal device 110 may pre-process the captured voice data to eliminate noise (such as ambient sound) in the voice data that is not related to the question.

[0052] Referring back to FIG. 2 , at block 220 , the terminal device 110 determines whether the image data contains potentially private information.

[0053] The terminal device 110 may detect the captured image data to determine whether the image data contains potential privacy information.

[0054] In some embodiments, the terminal device 110 can use a trained detection model to detect image data. The terminal device 110 inputs the image data into the detection model to obtain a detection result indicating whether it contains private information. The detection result can, for example, be a direct recognition result of an object in the image data, such as the name of the object contained in the image data. The object name output by the detection model can be, for example, an ID card, a water cup, a bank card, a bicycle, etc. The terminal device 110 can pre-acquire a library of names associated with potential private information and compare the object names in the image data output by the detection model with the names in the name library. If it is determined that an object name exists that is identical to a name in the name library, the terminal device 110 can determine that the image data contains potential private information. For example, if the object names output by the detection model include "bank card" and "water cup," and the terminal device 110 determines that the name "bank card" exists in the name library, it determines that the image data contains potential private information.

[0055] In some embodiments, to refine the detection results and improve detection accuracy, the terminal device 110 may divide the image data into different regions (these multiple regions may be referred to as multiple candidate regions). The terminal device 110 uses a trained detection model to determine the detection results corresponding to each of the multiple candidate regions. The detection result may be, for example, a privacy risk score. That is, the detection model may output a privacy risk score corresponding to the input candidate region. The terminal device 110 may then determine whether the corresponding candidate region contains potential privacy information based on a comparison of the privacy risk score and a threshold score. The threshold score may be a score preset by the user or a score determined by the terminal device 110 based on historical detection results. Specifically, if the privacy risk score is greater than or equal to the threshold score, the terminal device 110 may determine that the corresponding candidate region contains potential privacy information. If the privacy risk score is less than the threshold score, the terminal device 110 may determine that the corresponding candidate region does not contain potential privacy information. The terminal device 110 may determine that one or more candidate regions determined to contain potential privacy information are target regions.

[0056] In some embodiments, if it is determined that the image data does not contain potentially private information, the terminal device 110 may determine an answer to the question based on the image data and the question. For example, the terminal device 110 may use a trained question-answering model to determine an answer to the image data based on the image data and the question. It should be noted that if the question is captured in the form of voice, the terminal device 110 must first convert the voice data corresponding to the question into a text sequence, and then input the text sequence and the image data into the trained question-answering model so that the question-answering model outputs an answer corresponding to the question.

[0057] In some embodiments, since the computing power of the terminal device 110 is limited, in order to ensure the accuracy of the output results of the question-answering model, the trained question-answering model can be deployed in a remote device. The terminal device 110 can use the trained question-answering model to implement the question-answering function through communication with the remote device. In such a case, if it is determined that the image data does not contain potential privacy information, the terminal device 110 can determine that the captured image data will not cause the user's privacy information to be leaked, and then the image data can be sent to the remote device. After the remote device uses the trained question-answering model to determine the answer to the question, it sends the answer to the terminal device 110 so that the terminal device 110 determines the answer to the question.

[0058] If it is determined in block 220 that the image data contains potential private information, in block 230 , the terminal device 110 determines whether the question is an intentional question regarding the potential private information.

[0059] In an embodiment of the present disclosure, if the image data is determined to contain potentially private information, the terminal device 110 further identifies intentional questions regarding the potentially private information to determine whether the user intentionally wanted to capture the potentially private information in the image data and desired to ask questions regarding the potentially private information. This is because in a question-and-answer scenario based on image data, the image data is the basis for determining the answer to the question. Furthermore, there are special cases, such as assisting the visually impaired, where intentional questions regarding potentially private information are necessary. In such cases, it is necessary to support subsequent processing using image data containing potentially private information.

[0060] In some embodiments, the terminal device 110 can perform intent recognition on the question to determine the intent associated with the question. Based on the determined intent, the terminal device 110 then determines whether the question is an intentional question regarding potentially private information. Specifically, the terminal device 110 can identify intent based on a rule-based approach using dictionaries and templates. Different intents have different domain dictionaries, such as book titles, song titles, product names, object names, etc. The terminal device 110 can make a determination based on the degree of match or overlap between the user's intent and the dictionary. The terminal device 110 can also identify user intent based on a machine learning model. The terminal device 110 can train and learn from annotated domain corpus using machine learning and deep learning methods to obtain an intent recognition model (e.g., a model based on fastText). The terminal device 110 then identifies the intent indicated by the input question based on this model. It will be understood that the terminal device 110 can determine whether the question is an intentional question regarding potentially private information locally, or it can send the question to a remote device for the remote device to determine whether the question is an intentional question regarding potentially private information.

[0061] For example, when determining that the image data contains an object such as a bank card containing user private information, the terminal device 110 can determine that the image data contains potentially private information. The terminal device 110 then identifies the question entered by the user. If the question is a question about a bank card, such as "What is the card number?" or "Which bank's bank card is this?", the terminal device 110 can determine that the intent of the question is related to the bank card and, therefore, the question is an intentional question regarding potentially private information. If the question is a question about a bank card, such as "Is there a water cup in the image?" or "What color is the table?", the terminal device 110 can determine that the intent of the question is not related to the bank card and that the user's question is not an intentional question regarding potentially private information.

[0062] In block 240 , the terminal device 110 performs privacy protection processing on the image data based on the determination of whether the question is an intentional question regarding potentially private information.

[0063] In some embodiments, terminal device 110 may perform different privacy protection operations on the image data based on the result of determining whether the question is an intentional question regarding potentially private information. Specifically, terminal device 110 may perform privacy protection operations on a portion of the image data if it determines that the question is not an intentional question regarding potentially private information, and may not perform privacy protection operations on the image data if it determines that the question is an intentional question. For example, the privacy protection operation may include mosaicing the image data to cover it, deleting the area, etc., or any other operation that prevents the exposure of potentially private information.

[0064] In some embodiments, if it is determined that the question is not an intentional question regarding potentially private information, the user's question may be related to other objects in the image data (for example, when the image data contains a water cup and a bank card). In this case, to avoid performing a privacy protection operation on the entire image data and affecting other objects in the image data, which in turn affects subsequent question-and-answer results, the terminal device 110 may perform a privacy protection operation only on the area in the image data containing potentially private information.

[0065] In some embodiments, the terminal device 110 can use a trained image segmentation model to identify areas in the image data where potential private information is located. Thus, the terminal device 110 can perform privacy protection operations only on the identified areas when performing subsequent privacy protection operations.

[0066] In some embodiments, the terminal device 110 may further pre-divide the image data to determine multiple candidate regions contained in the image data. The terminal device 110 may determine the candidate regions containing potential privacy information as target regions and perform privacy protection operations only on the target regions.

[0067] Furthermore, the terminal device 110 may transmit the privacy-preserving processed image data to a remote device, such as the computing device 132, for the remote device to determine a response to the question. In some embodiments, the question captured by the terminal device 110 is also transmitted to the remote device, such as the computing device 132, along with the image data.

[0068] In some embodiments, if the question is determined to be an intentional question regarding potentially private information, and if the terminal device 110 determines that the potentially private information was intentionally captured by the user, the terminal device 110 may transmit the image data to a remote device, such as the computing device 132, for use in determining a response to the question, without performing any privacy protection operations on the image data. In some embodiments, the question captured by the terminal device 110 is also transmitted to the remote device, such as the computing device 132, along with the image data.

[0069] In some embodiments, to further reduce the risk of privacy leakage, the terminal device 110 may also provide a confirmation request to the user if it determines that the question is an intentional question about potential private information, requesting confirmation whether to allow the image data to be remotely processed. The terminal device 110 may provide the confirmation request to the user by, for example, playing a voice message (e.g., playing an audio message saying "The image contains XX potential private information, please confirm whether to continue the question and answer process"), presenting text (e.g., presenting a text prompt on the display), playing a video, vibrating, flashing the screen, or other one or more methods.

[0070] In some embodiments, while providing the confirmation request, the terminal device 110 may also provide a prompt message to the user, which may indicate that the image data will not be remotely stored. The terminal device 110 may provide the prompt message to the user by, for example, playing a voice message (e.g., playing an audio message stating "This image will not be remotely stored"), presenting text (e.g., presenting a text prompt on a display screen), playing a video, vibrating, flashing the screen, etc.

[0071] Furthermore, the terminal device 110 may transmit the image data to a remote device, such as the computing device 132, in response to receiving a confirmation indication for the confirmation request. The terminal device 110 may determine that the confirmation indication has been received, for example, in response to receiving a voice message (e.g., receiving a voice message saying "Confirm"). The terminal device 110 may also determine that the confirmation indication has been received, for example, in response to detecting a preset operation on a confirmation control or a preset operation on a hardware button 450.

[0072] Therefore, in the visual language question-answering scenario, the solution disclosed in the present invention can protect user privacy while enabling intentional questions regarding potential private information to be answered immediately. In the case of unintentional questions, privacy protection operations can be performed on image data immediately, thereby ensuring the efficiency of answers and avoiding unconscious privacy leaks of users.

[0073] In some embodiments, if it is determined that the image data contains potentially private information, to ensure data security, the terminal device 110 may set a save flag associated with the image data to prohibit remote saving. Setting the save flag to prohibit remote saving can instruct subsequent data processing devices not to save the image data. Setting this save flag can clearly indicate the special attributes of the current image data to the remote computing device, preventing privacy leaks caused by remote computing during subsequent data processing, thereby helping to protect the user's private information.

[0074] Referring to FIG. 4 , FIG. 4 illustrates a schematic diagram of a data processing process 400 according to some embodiments of the present disclosure. Process 400 may be implemented at terminal device 110 . For ease of discussion, process 400 will be described with reference to environment 100 of FIG. 1 . Note that image data 401 and question 402 shown in FIG. 4 are merely examples for illustrative purposes and are not intended to indicate any limitation.

[0075] In some embodiments, after receiving image data 401, terminal device 110 performs visual privacy information identification on image data 401 at block 410. At block 420, terminal device 110 determines whether image data 401 contains potential privacy information. If terminal device 110 determines that image data 401 does not contain potential privacy information, it sends image data 401 to computing device 132.

[0076] If it is determined that image data 401 contains potentially private information, terminal device 110 sets the save flag of image data 401 to prohibit remote storage at block 430. At block 440, terminal device 110 performs question recognition on question 402 (e.g., which may be in the form of speech). During question recognition, terminal device 110 may recognize the captured speech data as text and determine the intent of the question.

[0077] At block 450 , terminal device 110 determines whether question 402 is an intentional question regarding potentially private information. If it is determined that the question is not an intentional question regarding potentially private information, at block 460 , terminal device 110 performs a privacy protection operation on the region of image data 401 containing the potentially private information, such as erasing the region. Terminal device 110 then transmits the privacy-protected image data 401 to a remote computing device 132. In some embodiments, question 402 is also transmitted along with image data 401 to computing device 132.

[0078] If question 402 is determined to be an intentional question regarding potentially private information, terminal device 110 provides a confirmation request and / or prompt information to the user at block 470. Upon receiving a confirmation indication regarding the confirmation request, terminal device 110 may transmit image data 401 to remote computing device 132. In some embodiments, question 402 is also transmitted to computing device 132 along with image data 401.

[0079] The computing device 132 may output an answer to the image data 401 corresponding to the question 402 based on the received image data 401 and the question 402. The terminal device 110 may obtain the answer through communication with the computing device 132 and output the answer.

[0080] In some embodiments, after receiving the answer, the terminal device 110 can play the answer in voice form through a sound playback device. As shown in Figure 3, the terminal device 110 can play the answer audio through a speaker. In some embodiments, the answer can be in text form. The terminal device 110 can convert the text into speech for output through text-to-speech synthesis (TTS). This makes it convenient for users, especially those with visual impairments, to quickly obtain the answer.

[0081] In some embodiments, the terminal device 110 may alternatively present the answer in text form via the display screen. In some embodiments, the terminal device 110 may also output the answer in an additional vibrational or visual form. Visual forms may include, for example, magnified or highlighted images. For example, if the user inputs voice data indicating the name of an object in the query image data, the terminal device 110 may, while playing the audio answer containing the object's name, magnify the image data on the display screen to highlight the object.

[0082] In this way, image data can be differentiated, and based on the judgment of the question, privacy protection processing can only be performed on image data determined to contain potential privacy information, which helps reduce the risk of unintentional privacy leakage of users and thus improves the security of visual language question answering.

[0083] 5 shows a schematic structural block diagram of an apparatus 500 for data processing according to some embodiments of the present disclosure. Apparatus 500 may be implemented in or included in terminal device 110, for example. Each module / component in apparatus 500 may be implemented by hardware, software, firmware, or any combination thereof.

[0084] As shown, apparatus 500 includes a data capture module 510 configured to capture image data using a user's device and to instruct a question regarding the image data. Apparatus 500 also includes an information determination module 520 configured to determine whether the image data contains potentially private information. Apparatus 500 also includes a question determination module 530 configured to, if it is determined that the image data contains potentially private information, determine whether the question is an intentional question regarding the potentially private information. Apparatus 500 also includes a privacy protection processing module 540 configured to perform privacy protection processing on the image data based on the determination of whether the question is an intentional question regarding the potentially private information.

[0085] In some embodiments, the privacy protection processing module 540 includes: a regional privacy protection module, configured to perform a privacy protection operation on a region containing potential privacy information in the image data if it is determined that the question is not an intentional question regarding potential privacy information; and a first data sending module, configured to send the image data after the privacy protection processing to a remote device for determining an answer to the question.

[0086] In some embodiments, the privacy protection processing module 540 also includes: a question and answer processing module, which is configured to send the image data to a remote device for determining an answer to the question if it is determined that the question is an intentional question regarding potential private information, without performing privacy protection operations on the image data.

[0087] In some embodiments, the first data sending module includes: a confirmation request providing module, configured to provide a confirmation request to the user if it is determined that the question is an intentional question regarding potential private information, requesting confirmation as to whether the image data is allowed to be remotely processed; and an image data sending module, configured to send the image data to the remote device in response to receiving a confirmation indication for the confirmation request.

[0088] In some embodiments, the apparatus 500 further includes: a prompt information providing module configured to provide prompt information to the user to indicate that the image data will not be stored remotely.

[0089] In some embodiments, the apparatus 500 further includes: a save flag setting module configured to set a save flag associated with the image data to prohibit remote saving if it is determined that the image data contains potential privacy information.

[0090] In some embodiments, the apparatus 500 further includes: a second data sending module configured to send the image data to a remote device for determining an answer to the question if it is determined that the image data does not contain potential privacy information.

[0091] In some embodiments, the question comprises a question captured in speech form.

[0092] The units included in the device 500 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units can be implemented using software and / or firmware, such as machine executable instructions stored on a storage medium. In addition to or as an alternative to machine executable instructions, some or all of the units in the device 500 can be implemented at least in part by one or more hardware logic components. By way of example and not limitation, exemplary types of hardware logic components that can be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0093] FIG6 shows a block diagram of an electronic device 600 in which one or more embodiments of the present disclosure may be implemented. It should be understood that the electronic device 600 shown in FIG6 is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. The electronic device 600 shown in FIG6 can be used to implement the electronic device 110 of FIG1 .

[0094] As shown in FIG6 , electronic device 600 is a general-purpose electronic device. Components of electronic device 600 may include, but are not limited to, one or more processors or processing units 610, memory 620, storage device 630, one or more communication units 640, one or more input devices 650, and one or more output devices 660. Processing unit 610 may be a real or virtual processor and is capable of performing various processes according to programs stored in memory 620. In a multi-processor system, multiple processing units execute computer-executable instructions in parallel to enhance the parallel processing capabilities of electronic device 600.

[0095] The electronic device 600 typically includes a plurality of computer storage media. Such media can be any accessible media that can be obtained by the electronic device 600, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 620 can be a volatile memory (e.g., a register, a cache, a random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 630 can be a removable or non-removable medium and can include a machine-readable medium, such as a flash drive, a disk, or any other medium that can be used to store information and / or data (e.g., training data for training) and can be accessed within the electronic device 600.

[0096] The electronic device 600 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not shown in FIG6 , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a “floppy disk”) and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to a bus (not shown) by one or more data media interfaces. The memory 620 may include a computer program product 625 having one or more program modules configured to perform various methods or actions of various embodiments of the present disclosure.

[0097] The communication unit 640 enables communication with other electronic devices via a communication medium. Additionally, the functions of the components of the electronic device 600 can be implemented in a single computing cluster or multiple computing machines that can communicate via a communication connection. Thus, the electronic device 600 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0098] The input device 650 may be one or more input devices, such as a mouse, keyboard, or trackball. The output device 660 may be one or more output devices, such as a display, a speaker, or a printer. The electronic device 600 may also communicate with one or more external devices (not shown) through the communication unit 640 as needed, such as a storage device, a display device, or the like, with one or more devices that allow a user to interact with the electronic device 600, or with any device that allows the electronic device 600 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0099] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0100] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0101] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, such that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0102] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0103] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple implementations of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part for a module, program segment or instruction, and a part for a module, program segment or instruction comprises one or more executable instructions for realizing the logical function of the specification. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0104] While various implementations of the present disclosure have been described above, the foregoing description is intended to be illustrative, not exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is selected to best explain the principles of the implementations, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A data processing method, include: capturing image data using a user's device and indicating a question to be asked regarding the image data; determining whether the image data contains potential privacy information; If it is determined that the image data contains potential private information, determining whether the question is an intentional question regarding the potential private information; and Based on a determination of whether the question is an intentional question regarding the potentially private information, a privacy protection process is performed with respect to the image data.

2. The method according to claim 1, wherein a privacy protection process is performed on the image data. include: If it is determined that the question is not an intentional question regarding the potential private information, performing a privacy protection operation on the region of the image data containing the potential private information; as well as The image data after the privacy protection process is performed is sent to a remote device for determining an answer to the question.

3. The method according to claim 1 or 2, wherein performing privacy protection processing on the image data further comprises: include: If it is determined that the question is an intentional question regarding the potentially private information, the image data is sent to a remote device for determining an answer to the question without performing a privacy protection operation on the image data.

4. The method of claim 3, wherein the image data is sent to the remote device include: If it is determined that the question is an intentional question regarding the potential private information, providing a confirmation request to the user to request confirmation of whether to allow the image data to be remotely processed; as well as In response to receiving a confirmation indication for the confirmation request, the image data is sent to the remote device.

5. The method according to claim 3, further comprising: include: Prompt information is provided to the user to indicate that the image data will not be stored remotely.

6. The method according to claim 1, further comprising: include: If it is determined that the image data contains the potential privacy information, a save flag associated with the image data is set to prohibit remote saving.

7. The method according to claim 1, further comprising: include: If it is determined that the image data does not contain the potentially private information, the image data is sent to a remote device for use in determining a response to the question. The method of claim 1 , wherein the question comprises a question captured in speech form.

9. A device for data processing, include: A data capture module configured to capture image data using a user's device and to instruct questions regarding the image data; an information determination module, configured to determine whether the image data contains potential privacy information; a question determination module, configured to determine whether the question is an intentional question regarding the potential private information if it is determined that the image data contains potential private information; as well as The privacy protection processing module is configured to perform privacy protection processing on the image data based on the determination of whether the question is an intentional question regarding the potential private information.

10. The device according to claim 9, wherein the privacy protection processing module include: a regional privacy protection module, configured to perform a privacy protection operation on a region of the image data containing the potential private information if it is determined that the question is not an intentional question regarding the potential private information; as well as The first data sending module is configured to send the image data after the privacy protection process is performed to a remote device for determining an answer to the question.

11. The device according to claim 9 or 10, wherein the privacy protection processing module further include: The question-answer processing module is configured to send the image data to a remote device for determining an answer to the question without performing a privacy protection operation on the image data if it is determined that the question is an intentional question about the potential private information.

12. The device according to claim 11, wherein the first data sending module include: a confirmation request providing module, configured to provide a confirmation request to the user if it is determined that the question is an intentional question regarding the potential private information, so as to request confirmation as to whether to allow the image data to be remotely processed; as well as The image data sending module is configured to send the image data to the remote device in response to receiving a confirmation indication for the confirmation request.

13. The device according to claim 11, further comprising: include: The prompt information providing module is configured to provide prompt information to the user to indicate that the image data will not be stored remotely.

14. The device according to claim 9, further comprising: include: The saving flag setting module is configured to set a saving flag associated with the image data to prohibit remote saving if it is determined that the image data contains the potential privacy information.

15. The device according to claim 9, further comprising: include: The second data sending module is configured to send the image data to a remote device for determining an answer to the question if it is determined that the image data does not contain the potential privacy information.

16. The apparatus of claim 9, wherein the question comprises a question captured in speech form.

17. An electronic device, include: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 8 when executed by the at least one processing unit.

18. A computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the method according to any one of claims 1 to 8 when executed by a processor.

Citation Information

Patent Citations

  • Visual fog

    CN112384947A

  • Visual question and answer processing method and device, computer readable medium and program product

    CN113722458A

  • Digital human interaction method and device, electronic equipment and computer storage medium

    CN113849795A

  • Digital human interaction method and device, electronic equipment and computer storage medium

    CN113868401A

  • Multi-mode privacy sensitive data detection and encryption method

    CN114817951A

Cited By

  • Large model question and answer method and device capable of customizing personal privacy security protection

    CN119808167A