Method, apparatus, device and storage medium for data processing

Abstract

Embodiments of the present disclosure provide a method, apparatus, device and storage medium for data processing. The method comprises: capturing image data and a question directed to the image data by a device of a user; determining whether the image data contains potential privacy information; if it is determined that the image data contains potential privacy information, determining whether the question is an intentional question directed to the potential privacy information; and performing privacy protection processing for the image data based on the determination of whether the question is an intentional question directed to the potential privacy information. Thus, based on the judgment of whether it is an intentional privacy question, differentiated privacy protection processing can be performed on the image data, which helps to reduce the risk of unintentional privacy disclosure of the user and helps to provide instant question and answer services for the user in the case of intentional question.

Description

Technical Field

[0001] The exemplary embodiments disclosed herein relate generally to the field of computers, and more particularly to methods, apparatus, devices and computer-readable storage media for data processing. Background Technology

[0002] With the rapid development of information technology, more and more applications offer question-and-answer functionality, bringing numerous conveniences to users. Applications with question-and-answer capabilities can output corresponding answers based on user input of voice or text. Applications with multimodal visual question answering (VQA) capabilities can also output audio answers based on user-inputted images, asking questions via voice. However, it is also desirable to implement privacy protection measures for the input images while achieving multimodal visual question answering. Summary of the Invention

[0003] In a first aspect of this disclosure, a data processing method is provided. The method includes: capturing image data using a user's device and indicating a question to the image data; determining whether the image data contains potential privacy information; if the image data is determined to contain potential privacy information, determining whether the question is an intentional question regarding the potential privacy information; and performing privacy-preserving processing on the image data based on the determination that the question is an intentional question regarding the potential privacy information.

[0004] In a second aspect of this disclosure, an apparatus for data processing is provided. The apparatus includes: a data capture module configured to capture image data using a user's device and to instruct a question to be asked about the image data; an information determination module configured to determine whether the image data contains potential privacy information; a question determination module configured to determine whether, if the image data is determined to contain potential privacy information, the question is an intentional question regarding the potential privacy information; and a privacy protection processing module configured to perform privacy protection processing on the image data based on the determination that the question is an intentional question regarding the potential privacy information.

[0005] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the electronic device to perform the method of the first aspect.

[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program that can be executed by a processor to implement the method of the first aspect.

[0007] It should be understood that the description in this section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0008] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0009] Figure 1 A schematic diagram of an example environment in which embodiments of the present disclosure can be implemented is shown;

[0010] Figure 2 A flowchart illustrating a data processing procedure according to some embodiments of the present disclosure is shown;

[0011] Figure 3 A schematic diagram of a question-and-answer interface according to some embodiments of the present disclosure is shown;

[0012] Figure 4 A schematic diagram of a data processing flow according to some embodiments of the present disclosure is shown;

[0013] Figure 5 A schematic structural block diagram of an apparatus for data processing according to some embodiments of the present disclosure is shown; and

[0014] Figure 6 A block diagram of an electronic device that may implement one or more embodiments of the present disclosure is shown. Detailed Implementation

[0015] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0016] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below. As used herein, the term "model" can represent the relationship between various data. For example, the above-mentioned relationship can be obtained based on various technical solutions that are currently known and / or will be developed in the future.

[0017] In this document, unless explicitly stated otherwise, performing a step in response to A does not mean that the step is performed immediately after A, but may include one or more intermediate steps.

[0018] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0019] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure through appropriate means in accordance with relevant laws and regulations, and user authorization should be obtained.

[0020] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0021] As an optional but non-limiting embodiment, in response to receiving a user's active request, a prompt message can be sent to the user, for example, via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window may also include a target exploration control allowing the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0022] It is understood that the above notification and user authorization acquisition process is merely illustrative and does not constitute a limitation on the embodiments of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the embodiments of this disclosure.

[0023] As used in this paper, the term "model" refers to a model that learns the relationship between inputs and outputs from training data, enabling it to generate corresponding outputs for a given input after training. Model generation can be based on machine learning techniques. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs using multiple layers of processing units. A neural network model is an example of a deep learning-based model. In this paper, "model" may also be referred to as a "machine learning model," "learning model," "machine learning network," or "learning network," and these terms are used interchangeably.

[0024] A neural network is a machine learning network based on deep learning. A neural network processes input and provides a corresponding output, typically consisting of an input layer, an output layer, and one or more hidden layers between the input and output layers. Neural networks used in deep learning applications often include many hidden layers, thus increasing the network's depth. The layers of a neural network are connected sequentially, so that the output of the previous layer is provided as the input to the next layer. The input layer receives the input to the neural network, while the output layer's output serves as the final output. Each layer of a neural network includes one or more nodes (also called processing nodes or neurons), each node processing the input from the layer above.

[0025] Machine learning typically comprises three phases: training, testing, and application (also known as inference). In the training phase, a given model is trained using a large amount of training data, iteratively updating its parameter values ​​until the model can consistently generate inferences that meet the expected goals from the training data. Through training, the model can be considered to have learned the relationship between inputs and outputs (also known as the input-output mapping) from the training data. The parameter values ​​of the trained model are determined. In the testing phase, test inputs are applied to the trained model to test whether it can provide the correct output, thus determining the model's performance. In the application phase, the model can be used to process actual inputs based on the trained parameter values ​​to determine the corresponding output.

[0026] As briefly mentioned earlier, applications with multimodal visual language question answering capabilities can play audio responses to images based on user-inputted voice questions.

[0027] Visual language question answering (VLE) is a multimodal understanding task that requires understanding visual content to answer verbal questions. Traditionally, applications with multimodal VLE capabilities have utilized trained question-answering models to achieve multimodal VLE functionality.

[0028] However, people may sometimes fail to notice whether their captured image data contains private information they do not wish to disclose. This poses a risk of unintentional privacy breaches in visual-based question-answering scenarios. Specifically, the process of a question-answering model determining the answer may be performed remotely (e.g., in the cloud). Furthermore, question-answering models require the collection of large amounts of image data for updates or improvements, which is then provided to annotators for annotation. All of this can lead to the leakage of unintentionally captured private information from the image data. This is particularly true for certain groups, such as the visually impaired, who have a greater need for visual-language question answering. Due to their visual limitations, these individuals often find it more difficult to detect whether their captured image data contains private information they do not wish to disclose. Therefore, there is a desire to achieve better privacy protection while providing question-answering services in visual-based question-answering scenarios.

[0029] Embodiments of this disclosure propose an improved scheme for data processing. This scheme relates to question-and-answer based on image data (visual data). According to this scheme, image data is collected, and a question is indicated regarding the image data. If it is determined that the image data contains potential privacy information, it is judged whether the question is an intentional question targeting that potential privacy information. Based on the judgment result, privacy-preserving processing is performed on the image data. In this way, discriminative privacy-preserving processing can be performed on the image data based on the judgment of whether it is an intentional privacy question. This helps reduce the risk of unintentional privacy leakage by users and improves security in the context of question-and-answer based on visual information. Furthermore, it can help provide users with immediate question-and-answer services in the event of an intentional question targeting potential privacy information.

[0030] Furthermore, the question-answering scheme proposed in this disclosure can effectively assist users, especially those with persistent or temporary visual impairment or disability, in safely and reliably achieving multimodal visual-verbal question answering. It should be understood that the solutions provided by the embodiments of this disclosure may offer convenience to specific groups, but this does not imply any discrimination against any particular group.

[0031] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. In this example environment 100, an application 120 is installed on a terminal device 110. A user 140 can interact with the application 120 via the terminal device 110 and / or its attached devices. The application 120 is an application that has at least visual language question-and-answer functionality.

[0032] In some embodiments, terminal device 110 communicates with computing device 132 in cloud computing environment 130 to provide services to application 120. Terminal device 110 can be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some embodiments, terminal device 110 can also support any type of user-facing interface (such as "wearable" circuitry).

[0033] Terminal device 110 may include, for example, appropriate types of sensors for detecting user gestures. For instance, terminal device 110 may include a touchscreen for detecting various types of gestures made by the user on the touchscreen. Alternatively or additionally, terminal device 110 may also include other appropriate types of sensing devices, such as proximity sensors, to detect various types of gestures made by the user within a predetermined distance above the screen. Terminal device 110 may also include, for example, sound acquisition devices (e.g., microphones) for acquiring user audio, sound playback devices (e.g., speakers) for playing audio, image acquisition devices (e.g., cameras, webcams, etc.) for acquiring images, and display devices (e.g., displays, which may be touchscreens) for displaying an interface.

[0034] In the cloud computing environment 130, the computing device 132 is a remote device relative to the terminal device 110. The computing device 132 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and big data and artificial intelligence platforms. The computing device 132 may include, for example, computing systems / servers such as mainframes, edge computing nodes, and computing devices in a cloud environment. The computing device 132 can provide backend services for the application 120 in the terminal device 110.

[0035] In some embodiments discussed below, visual language question answering functionality can be implemented using multiple models with various functions. One or more of these models can be remotely deployed in computing device 132, and terminal device 110 can utilize these multiple models to implement corresponding functions through communication with computing device 132 in cloud computing environment 130. This saves resources and power on terminal device 110 and allows for improved computing efficiency by leveraging the powerful resources of the server. In some embodiments, one or more of these models can also be deployed locally on terminal device 110. This can be chosen based on the specific circumstances.

[0036] In some embodiments, Figure 1 In environment 100, if application 120 is active, terminal device 110 can display the interface 150 of application 120. Through interface 150, application 120 can provide user 140 with one or more services related to the question-and-answer function, including capturing voice, capturing images, playing voice, displaying text, etc.

[0037] It should be understood that the structure and function of environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0038] The following description will continue with reference to the accompanying drawings, which will provide some exemplary embodiments of this disclosure.

[0039] Figure 2 A flowchart of a data processing procedure 200 according to some embodiments of the present disclosure is shown. Procedure 200 can be implemented at terminal device 110. For ease of discussion, reference will be made to... Figure 1 The environment 100 is used to describe the process 200.

[0040] In box 210, terminal device 110 captures image data and indicates questions regarding the image data.

[0041] In some embodiments, terminal device 110 may capture image data and indicate questions regarding the image data in response to detecting a question-and-answer initiation operation. Specifically, terminal device 110 may directly detect a user-initiated question-and-answer initiation operation. For example, terminal device 110 may determine that a question-and-answer initiation operation has been detected in response to detecting a question-and-answer initiation voice (e.g., "Open the question-and-answer function"). Alternatively, terminal device 110 may determine that a question-and-answer initiation operation has been detected in response to detecting a preset operation on a hardware button (e.g., a press operation, a long press operation, etc.). In some embodiments, after detecting a question-and-answer initiation operation, terminal device 110 runs application 120 with question-and-answer functionality, captures image data, and indicates questions regarding the image data.

[0042] Terminal device 110 can capture image data through an image acquisition device. The image data can be in any form (still image, video clip, etc.), at any resolution, and in any format (e.g., PNG, JPG, etc.). Alternatively or additionally, the image data can also be data pre-stored in terminal device 110.

[0043] Questions can take many forms, meaning the terminal device 110 can capture questions in various ways. For example, questions can be captured in text form, where the terminal device 110 can directly acquire the text sequence of the user's input question (e.g., acquiring the text sequence "How many cups are in the image?"). To ensure convenience and ease of operation, questions can also be captured in voice form, where the terminal device 110 can capture the user's voice data through a sound acquisition device. The voice data can be in any language (e.g., Chinese, English, Japanese, etc.), of any duration (e.g., 3s, 5s, etc.), and with any timbre. It is understood that questions can also be captured in any other suitable form.

[0044] In cases where the question is captured in voice format, in some embodiments, after the terminal device 110 launches the application 120, it can present a recording interface, including at least a recording control, through a display device. The terminal device 110 can determine that a question-and-answer initiation operation has been detected in response to detecting a predetermined operation on the recording control, and then capture image data and voice data. The predetermined operation on the recording control may include, for example, a click operation, a swipe operation, a long press operation, etc., and is not limited thereto. In some embodiments, the predetermined operation on the recording control can also be initiated via voice or other commands.

[0045] In some embodiments, during the capture of image and audio data, the terminal device 110 may also stop capturing image and audio data in response to receiving a capture end operation. Specifically, the terminal device 110 may determine that a capture end operation has been detected in response to detecting, for example, an audio message saying "Stop capturing data". The terminal device 110 may also determine that a capture end operation has been detected in response to detecting a preset operation on a hardware button (e.g., a press operation, a long press operation, etc.). The terminal device 110 may also determine that a capture end operation has been detected in response to detecting another predetermined operation on a recording control in the recording interface (e.g., a click operation, a release operation, etc.).

[0046] refer to Figure 3 , Figure 3A schematic diagram of a recording interface 300 according to some embodiments of the present disclosure is shown. The recording interface 300 may include a control display area 330, which at least displays a recording control 332. The terminal device 110 may determine that a question-and-answer initiation operation has been detected in response to detecting a predetermined operation on the recording control 332. The terminal device 110 may then display text such as "Recording" in a text prompt area 310 to indicate to the user that the terminal device 110 is currently in a data capture state.

[0047] In some embodiments, in response to receiving a predetermined operation, the terminal device 110 indicates that it is capturing audio and image data by changing the presentation of the recording control 332 (e.g., changing the color, size, etc. of the recording control 332). Accordingly, the terminal device 110 may stop capturing audio and image data in response to receiving a capture end operation. The terminal device 110 may also switch the presentation of the recording control 332 back to its state before capture in response to receiving a capture end operation.

[0048] In some embodiments, the terminal device 110 can convert the captured voice data into text and display it in the text prompt area 310. For example... Figure 3 As shown, after capturing voice data, the terminal device 110 displays the text "What is the card number?" corresponding to the voice data in the text prompt area 310. Simultaneously, the terminal device 110 can display the image data currently captured by the terminal device 110 in the image display area 320. The speech-to-text conversion can be achieved using speech-to-text technology, and this conversion can be performed locally on the terminal device 110 or on a remote server.

[0049] In some embodiments, to ensure the accuracy of the question indicated by the determined voice data, the terminal device 110 may preprocess the captured voice data to eliminate noise (e.g., ambient noise) in the voice data that is irrelevant to the question.

[0050] Return to reference Figure 2 In box 220, terminal device 110 determines whether the image data contains potential privacy information.

[0051] Terminal device 110 can detect captured image data to determine whether the image data contains potential privacy information.

[0052] In some embodiments, terminal device 110 can utilize a trained detection model to detect image data. Terminal device 110 inputs image data into the detection model to obtain a detection result indicating whether it contains privacy information. The detection result can be, for example, a direct identification of objects in the image data, such as the name of an object contained in the image data. The object name output by the detection model can be, for example, an ID card, a water cup, a bank card, a bicycle, etc. Terminal device 110 can pre-obtain a name database associated with potential privacy information and compare the object names in the image data output by the detection model with the names in the name database. If it is determined that an object name exists that is identical to a name in the name database, terminal device 110 can determine that the image data contains potential privacy information. For example, if the object names output by the detection model include "bank card" and "water cup," and terminal device 110 determines that the image data contains potential privacy information if it determines that the name "bank card" exists in the name database.

[0053] In some embodiments, to refine the detection results and improve detection accuracy, the terminal device 110 can divide the image data into different regions (these multiple regions can be referred to as multiple candidate regions). The terminal device 110 uses a trained detection model to determine the detection results corresponding to each of these multiple candidate regions. The detection results can be, for example, a privacy risk score. That is, the detection model can output a privacy risk score corresponding to the input candidate region. The terminal device 110 can then determine whether the corresponding candidate region contains potential privacy information based on a comparison between the privacy risk score and a threshold score. The threshold score can be a score preset by the user or a score determined by the terminal device 110 based on historical detection results. Specifically, if the privacy risk score is greater than or equal to the threshold score, the terminal device 110 can determine that the corresponding candidate region contains potential privacy information. If the privacy risk score is less than the threshold score, the terminal device 110 can determine that the corresponding candidate region does not contain potential privacy information. The terminal device 110 can determine one or more candidate regions containing potential privacy information as target regions.

[0054] In some embodiments, if it is determined that the image data does not contain potential privacy information, the terminal device 110 can determine an answer to the question based on the image data and the question. For example, the terminal device 110 can utilize a trained question-answering model to determine an answer to the image data based on the image data and the question. It should be noted that when the question is captured in the form of speech, the terminal device 110 needs to first convert the speech data corresponding to the question into a text sequence, and then input the text sequence along with the image data into the trained question-answering model so that the question-answering model can output an answer corresponding to the question.

[0055] In some embodiments, since the computing power of the terminal device 110 is limited, the trained question-answering model can be deployed in a remote device to ensure the accuracy of the output results. The terminal device 110 can utilize the trained question-answering model to implement question-answering functionality through communication with the remote device. In such a case, if it is determined that the image data does not contain potential privacy information, the terminal device 110 can determine that the captured image data will not cause leakage of the user's privacy information, and can then send the image data to the remote device. After the remote device uses the trained question-answering model to determine the answer to the question, it sends the answer to the terminal device 110 so that the terminal device 110 can determine the answer to the question.

[0056] If it is determined in box 220 that the image data contains potential privacy information, in box 230, the terminal device 110 determines whether the question is an intentional question regarding the potential privacy information.

[0057] In embodiments of this disclosure, when it is determined that the image data contains potential privacy information, the terminal device 110 further identifies intentional questions regarding the potential privacy information to determine whether the potential privacy information is information that the user intends to capture in the image data and expects to ask questions about it. This is because in image data-based question-and-answer scenarios, the image data is the basis for determining the answer to the question. Additionally, there are special cases, such as when assisting visually impaired individuals, where intentional questions about potential privacy information are necessary. In such cases, it is necessary to support subsequent processing using image data containing potential privacy information.

[0058] In some embodiments, terminal device 110 can perform intent recognition on the question to determine the intent related to the question. Terminal device 110 then determines whether the question is intentionally asking about potential privacy information based on the determined intent. Specifically, terminal device 110 can identify intent based on dictionary and template rule-based methods. Different intents will have different domain dictionaries, such as book titles, song titles, product names, object names, etc. Terminal device 110 can make a judgment based on the degree of matching or overlap between the user's intent and the dictionary. Terminal device 110 can also discriminate user intent based on machine learning models. Terminal device 110 can train and learn on pre-annotated domain corpora using machine learning and deep learning methods to obtain an intent recognition model (e.g., a fastText-based model). Terminal device 110 then identifies the intent indicated by the input question based on this model. It can be understood that terminal device 110 can determine whether the question is intentionally asking about potential privacy information locally, or it can send the question to a remote device for the remote device to determine whether the question is intentionally asking about potential privacy information.

[0059] For example, when determining that image data contains objects containing user private information, such as bank cards, terminal device 110 can determine that the image data contains potential privacy information. Terminal device 110 then identifies the user's input question. If the question is a bank card-related question such as "What is the card number?" or "Which bank is this card from?", terminal device 110 can determine that the intent of the question is related to bank cards, and thus determine that the question is an intentional question regarding potential privacy information. If the question is not related to bank cards, such as "Is there a water glass in the image?" or "What color is the table?", terminal device 110 can determine that the intent of the question is unrelated to bank cards, and terminal device 110 can determine that the user's question is not an intentional question regarding potential privacy information.

[0060] In box 240, terminal device 110 performs privacy protection processing on image data based on the determination of whether the question is an intentional question targeting potential privacy information.

[0061] In some embodiments, the terminal device 110 may perform different privacy protection processes on the image data based on the result of determining whether the question is an intentional question targeting potential privacy information. Specifically, the terminal device 110 may perform privacy protection operations on a portion of the image data if it is determined that the question is not an intentional question targeting potential privacy information, and may not perform privacy protection operations on the image data if it is determined that the question is an intentional question. Privacy protection operations may include, for example, overlaying the image data by adding a mosaic, deleting the area, etc., and may also be any other operations to avoid exposing potential privacy information.

[0062] In some embodiments, where it is determined that the question is not an intentional question targeting potential privacy information, the user's question may be related to other objects in the image data (e.g., in the case where the image data contains a water glass and a bank card). In this case, to avoid affecting other objects in the image data due to privacy protection operations on the entire image data, which in turn could affect the subsequent question-and-answer results, the terminal device 110 may perform privacy protection operations only on the area of ​​the image data containing potential privacy information.

[0063] In some embodiments, the terminal device 110 may use a trained image segmentation model to identify regions containing potential privacy information from the image data. Therefore, the terminal device 110 can perform privacy protection operations only on the identified regions during subsequent privacy protection operations.

[0064] In some embodiments, the terminal device 110 may further pre-divide the image data to determine multiple candidate regions contained in the image data. The terminal device 110 may determine the candidate regions containing potential privacy information as target regions and perform privacy protection operations only on the target regions.

[0065] Furthermore, the terminal device 110 may send the image data, after privacy protection processing, to a remote device, such as computing device 132, for the remote device to determine an answer to the question. In some embodiments, the question captured by the terminal device 110 is also sent to the remote device, such as computing device 132, along with the image data.

[0066] In some embodiments, if it is determined that the question is an intentional question concerning potential privacy information, and if the terminal device 110 determines that the potential privacy information is information consciously captured by the user, the terminal device 110 may send the image data to a remote device, such as computing device 132, for determining an answer to the question, without performing any privacy protection operations on the image data. In some embodiments, the question captured by the terminal device 110 is also sent to a remote device, such as computing device 132, along with the image data.

[0067] In some embodiments, to further reduce the risk of privacy breaches, terminal device 110 may also provide a confirmation request to the user if it determines that the question is an intentional question targeting potential privacy information, requesting confirmation of whether to allow the image data to be processed remotely. Terminal device 110 may provide the confirmation request to the user, for example, by playing voice (e.g., playing an audio message such as "The image contains XX potential privacy information; please confirm whether to continue the Q&A"), presenting text (e.g., presenting a text prompt on the display screen), playing video, vibrating, or flashing the screen, etc.

[0068] In some embodiments, while providing a confirmation request, the terminal device 110 may also provide a prompt message to the user, indicating that the image data will not be remotely saved. The terminal device 110 may provide the prompt message to the user, for example, by playing voice (e.g., playing an audio message saying "This image will not be remotely saved"), displaying text (e.g., displaying a text prompt on the display screen), playing video, vibrating, or flashing the screen.

[0069] Furthermore, terminal device 110 may send image data to a remote device, such as computing device 132, in response to receiving a confirmation instruction for a confirmation request. Terminal device 110 may determine that a confirmation instruction has been received, for example, in response to receiving voice (e.g., receiving a voice message saying "confirm"). Terminal device 110 may also determine that a confirmation instruction has been received, for example, in response to detecting a preset operation on a confirmation control or a preset operation on a hardware button.

[0070] Therefore, in visual language question answering scenarios, the solution disclosed herein can protect user privacy while enabling intentional questions regarding potential privacy information to be answered instantly, and in the case of unintentional questions, it can perform privacy protection operations on image data in real time, thus ensuring the efficiency of the answer and preventing unintentional privacy leaks by users.

[0071] In some embodiments, when it is determined that the image data includes potentially private information, to ensure data security, the terminal device 110 may set a save flag associated with the image data to prohibit remote saving. Setting the save flag to prohibit remote saving instructs the device performing subsequent data processing not to save the image data. By setting this save flag, the special attributes of the current image data can be explicitly indicated to the remote computing device, avoiding privacy leaks caused by remote computing during subsequent data processing, and helping to protect the user's privacy information.

[0072] refer to Figure 4 , Figure 4 A schematic diagram of a data processing flow 400 according to some embodiments of the present disclosure is shown. Flow 400 can be implemented at terminal device 110. For ease of discussion, reference will be made to... Figure 1 The environment is 100 to describe process 400. Note that... Figure 4 The image data 401 and question 402 shown are merely illustrative examples and are not intended to indicate any limitations.

[0073] In some embodiments, after acquiring image data 401, the terminal device 110 performs visual privacy information recognition on the image data 401 in block 410. In block 420, the terminal device 110 determines whether the image data 401 contains potential privacy information. If the terminal device 110 then determines that the image data 401 does not contain potential privacy information, it sends the image data 401 to the computing device 132.

[0074] If it is determined that image data 401 contains potential privacy information, in box 430, terminal device 110 sets the save flag of image data 401 to prohibit remote save. In box 440, terminal device 110 performs question recognition on question 402 (e.g., which may be in the form of speech). During question recognition, terminal device 110 can recognize the captured speech data as text and determine the intent of the question.

[0075] In box 450, terminal device 110 determines whether question 402 is an intentional question regarding potential privacy information. If it is determined that the question is not an intentional question regarding potential privacy information, in box 460, terminal device 110 performs a privacy protection operation on the area of ​​image data 401 containing potential privacy information, such as erasing that area. Terminal device 110 then sends the privacy-protected image data 401 to a remote computing device 132. In some embodiments, question 402 is also sent to computing device 132 along with image data 401.

[0076] If it is determined that question 402 is an intentional question regarding potential privacy information, at box 470, terminal device 110 provides the user with a confirmation request and / or prompt information. Upon receiving a confirmation instruction in response to the confirmation request, terminal device 110 may send image data 401 to a remote computing device 132. In some embodiments, question 402 is also sent to computing device 132 along with image data 401.

[0077] The computing device 132 can output an answer to the image data 401 corresponding to the question 402 based on the received image data 401 and the question 402. The terminal device 110 can obtain the answer and output the answer through communication with the computing device 132.

[0078] In some embodiments, after receiving a response, the terminal device 110 can play the response in speech form via a sound playback device. For example... Figure 3 As shown, terminal device 110 can play the answer audio through a speaker. In some embodiments, the answer can be in text form. Terminal device 110 can convert text into speech and output it via text-to-speech (TTS). This allows users, especially visually impaired users, to quickly obtain the answer.

[0079] In some embodiments, alternatively, the terminal device 110 may also present the answer in text form via a display screen. In some embodiments, the terminal device 110 may also output the answer in both vibration and visual forms. Visual forms may include, for example, magnifying an image, highlighting an image, etc. For example, when a user inputs voice data indicating a query for the name of an object in image data, the terminal device 110 may magnify the image data on the display screen to highlight the object while playing an audio answer containing the name of the object.

[0080] In this way, image data can be differentiated, and based on the judgment of the question, privacy protection processing can be performed only on image data that is determined to contain potential privacy information. This helps to reduce the risk of users' unintentional privacy leakage, thereby improving the security of visual language question answering.

[0081] Figure 5 A schematic structural block diagram of an apparatus 500 for data processing according to some embodiments of the present disclosure is shown. The apparatus 500 may be implemented in or included in a terminal device 110, for example. The various modules / components in the apparatus 500 may be implemented by hardware, software, firmware, or any combination thereof.

[0082] As shown in the figure, device 500 includes a data capture module 510 configured to capture image data using a user's device and to prompt for questions regarding the image data. Device 500 also includes an information determination module 520 configured to determine whether the image data contains potential privacy information. Device 500 further includes a question determination module 530 configured to determine whether, if the image data is determined to contain potential privacy information, the question is an intentional question regarding the potential privacy information. Device 500 also includes a privacy protection processing module 540 configured to perform privacy protection processing on the image data based on the determination that the question is an intentional question regarding potential privacy information.

[0083] In some embodiments, the privacy protection processing module 540 includes: a region privacy protection module configured to perform a privacy protection operation on a region of image data containing potential privacy information if it is determined that the question is not an intentional question targeting potential privacy information; and a first data sending module configured to send the image data after privacy protection processing to a remote device for determining an answer to the question.

[0084] In some embodiments, the privacy protection processing module 540 further includes a question-and-answer processing module configured to send image data to a remote device for determining an answer to the question if it is determined that the question is an intentional question targeting potential privacy information, without performing privacy protection operations on the image data.

[0085] In some embodiments, the first data sending module includes: a confirmation request providing module configured to provide a confirmation request to a user if it is determined that the question is an intentional question concerning potential privacy information, in order to request confirmation as to whether image data is allowed to be processed remotely; and an image data sending module configured to send image data to a remote device in response to receiving a confirmation instruction for the confirmation request.

[0086] In some embodiments, the device 500 further includes a prompt information providing module configured to provide a prompt information to the user indicating that the image data will not be remotely saved.

[0087] In some embodiments, the apparatus 500 further includes a save flag setting module configured to set a save flag associated with the image data to prohibit remote saving if it is determined that the image data contains potential privacy information.

[0088] In some embodiments, the apparatus 500 further includes a second data transmission module configured to transmit the image data to a remote device for determining an answer to a question if it is determined that the image data does not contain potential privacy information.

[0089] In some embodiments, the question includes a question captured in voice form.

[0090] The units included in device 500 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units may be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the units in device 500 may be implemented at least partially by one or more hardware logic components. By way of example and not limitation, exemplary types of hardware logic components that may be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-chips (SoCs), complex programmable logic devices (CPLDs), and so on.

[0091] Figure 6 A block diagram of an electronic device 600 in which one or more embodiments of the present disclosure may be implemented is shown. It should be understood that... Figure 6 The electronic device 600 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. Figure 6 The electronic device 600 shown can be used to achieve Figure 1 Electronic devices 110.

[0092] like Figure 6 As shown, electronic device 600 is in the form of a general-purpose electronic device. Components of electronic device 600 may include, but are not limited to, one or more processors or processing units 610, memory 620, storage device 630, one or more communication units 640, one or more input devices 650, and one or more output devices 660. Processing unit 610 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 620. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 600.

[0093] Electronic device 600 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 600, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 620 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 630 can be a removable or non-removable medium and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data (e.g., training data for training) and can be accessed within electronic device 600.

[0094] Electronic device 600 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 6 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 620 may include computer program product 625 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0095] The communication unit 640 enables communication with other electronic devices via a communication medium. Additionally, the functionality of the components of the electronic device 600 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 600 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0096] Input device 650 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 660 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 600 can also communicate with one or more external devices (not shown) via communication unit 640 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 600, or with any device that enables electronic device 600 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0097] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0098] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0099] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0100] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0101] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0102] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A data processing method comprising: capturing image data and a question directed to the image data with a device of a user; determining whether the image data contains potential private information; if it is determined that the image data contains potential private information, determining whether the question is an intentional question directed to the potential private information; and based on the determination of whether the question is an intentional question directed to the potential private information, performing a privacy protection process for the image data, wherein determining whether the question is an intentional question directed to the potential private information comprises: performing intent recognition on the question to determine an intent related to the question; in response to determining that the intent is related to the potential private information, determining that the question is an intentional question directed to the potential private information; and in response to determining that the intent is not related to the potential private information, determining that the question is not an intentional question directed to the potential private information; wherein performing a privacy protection process for the image data comprises: if it is determined that the question is not an intentional question directed to the potential private information, performing a privacy protection operation on a region of the image data containing the potential private information, and based on the image data after performing the privacy protection process, determining an answer to the question, or if it is determined that the question is an intentional question directed to the potential private information, not performing a privacy protection operation on the image data, determining an answer to the question based on the image data.

2. The method of claim 1, further comprising: sending the image data after performing the privacy protection process to a remote device for determining an answer to the question.

3. The method of claim 1, wherein performing a privacy protection process for the image data further comprises: if it is determined that the question is an intentional question directed to the potential private information, sending the image data to a remote device for determining an answer to the question without performing a privacy protection operation on the image data.

4. The method of claim 3, wherein sending the image data to the remote device comprises: if it is determined that the question is an intentional question directed to the potential private information, providing a confirmation request to the user to request a confirmation of whether the image data is allowed to be processed remotely; and in response to receiving a confirmation indication for the confirmation request, sending the image data to the remote device.

5. The method of claim 3, further comprising: providing prompt information to the user to indicate that the image data will not be saved remotely.

6. The method of claim 1, further comprising: if it is determined that the image data contains the potential private information, setting a save flag associated with the image data to prohibit remote saving.

7. The method of claim 1, further comprising: if it is determined that the image data does not contain the potential private information, sending the image data to a remote device for determining an answer to the question. ​ 8. The method of claim 1, wherein the question comprises a question captured in voice form.

9. An apparatus for data processing, comprising: a data capturing module configured to capture image data and a question indicative of the image data with a device of a user; an information determining module configured to determine whether the image data contains potential private information; a question determining module configured to determine whether the question is an intentional question for the potential private information if it is determined that the image data contains potential private information; and a privacy protection processing module configured to perform a privacy protection processing for the image data based on the determination of whether the question is an intentional question for the potential private information, wherein the question determining module is further configured to: perform an intent recognition on the question to determine an intent related to the question; determine that the question is an intentional question for the potential private information in response to determining that the intent is related to the potential private information; and determine that the question is not an intentional question for the potential private information in response to determining that the intent is not related to the potential private information; wherein the privacy protection processing module comprises: a region privacy protection module configured to perform a privacy protection operation on a region of the image data containing the potential private information if it is determined that the question is not an intentional question for the potential private information, and determine an answer to the question based on the image data after performing the privacy protection processing, or an answer processing module configured to determine an answer to the question based on the image data if it is determined that the question is an intentional question for the potential private information, without performing a privacy protection operation on the image data.

10. The apparatus of claim 9, further comprising: a first data sending module configured to send the image data after performing the privacy protection processing to a remote device for determining an answer to the question.

11. The apparatus of claim 10, wherein the answer processing module is further configured to send the image data to a remote device for determining an answer to the question without performing a privacy protection operation on the image data if it is determined that the question is an intentional question for the potential private information.

12. The apparatus of claim 11, wherein the first data sending module comprises: a confirmation request providing module configured to provide a confirmation request to the user to request a confirmation of whether the image data is allowed to be processed remotely if it is determined that the question is an intentional question for the potential private information; and an image data sending module configured to send the image data to the remote device in response to receiving a confirmation indication for the confirmation request.

13. The apparatus of claim 11, further comprising: a prompt information providing module configured to provide prompt information to the user to indicate that the image data will not be saved remotely.

14. The apparatus of claim 9, further comprising: ​ ​ ​ A save flag setting module configured to set a save flag associated with the image data to prohibit remote saving if it is determined that the image data contains the potential private information.

15. The apparatus of claim 9, further comprising: a second data sending module configured to send the image data to a remote device for determining an answer to the question if it is determined that the image data does not contain the potential private information.

16. The apparatus of claim 9, wherein the question comprises a question captured in a voice form.

17. An electronic device, comprising: at least one processing unit; and at least one memory that is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit, the instructions, when executed by the at least one processing unit, cause the electronic device to perform the method according to any one of claims 1-8.

18. A computer readable storage medium having stored thereon a computer program, the computer program, when executed by a processor, implement the method according to any one of claims 1-8.

Images