Methods, apparatuses, devices, and media for processing sensitive information in visual data

Abstract

According to implementations of the present disclosure, methods, apparatuses, devices, and media for processing sensitive information in visual data are provided. In one method, a plurality of view images used to generate visual data associated with a target viewpoint are obtained. For a view image of the plurality of view images, a sensitive range of sensitive information in the view image is determined. A portion within the sensitive range in the view image is updated with a mask image. The visual data associated with the target viewpoint is generated based on the updated view image and other view images of the plurality of view images other than the view image. In this way, sensitive information can be processed in the process of producing visual data, and production efficiency can be improved while ensuring the elimination of sensitive information.

Description

TECHNICAL FIELD

[0001] Exemplary implementations of the present disclosure generally relate to visual data processing, and in particular, to a method, an apparatus, an equipment and a computer readable storage medium for processing sensitive information in visual data. BACKGROUND

[0002] With the development of digital technology, various types of visual data can be provided. For example, a video sequence can be shot and played; for another example, a panoramic image can be generated by using a plurality of images shot, and then a roaming in the panoramic scene can be performed. It will be understood that sensitive information such as a person, a text, etc. can exist in a shooting environment. If the original images collected are directly used to generate visual data without any desensitization processing, sensitive data can be included in the visual data. At this time, how to process the sensitive information in the visual data in a more convenient and fast manner becomes a problem to be solved. SUMMARY

[0003] In a first aspect of the present disclosure, a method for processing sensitive information in visual data is provided. In the method, a plurality of view images used to generate visual data associated with a target viewpoint are acquired. For a view image in the plurality of view images, a sensitive range of sensitive information in the view image is determined. A part within the sensitive range in the view image is updated by using a mask image. The visual data associated with the target viewpoint is generated based on the updated view image and other view images except the view image in the plurality of view images.

[0004] In a second aspect of the present disclosure, an apparatus for processing sensitive information in visual data is provided. The apparatus comprises: an acquisition module configured to acquire a plurality of view images used to generate visual data associated with a target viewpoint; a determination module configured to determine, for a view image in the plurality of view images, a sensitive range of sensitive information in the view image; an update module configured to update a part within the sensitive range in the view image by using a mask image; and a generation module configured to generate the visual data associated with the target viewpoint based on the updated view image and other view images except the view image in the plurality of view images.

[0005] In a third aspect of the present disclosure, an electronic device is provided. The electronic device comprises: at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions, when executed by the at least one processing unit, causing the device to perform the method according to the first aspect of the present disclosure.

[0006] In a fourth aspect of the present disclosure, a computer readable storage medium is provided, having stored thereon a computer program which, when executed by a processor, causes the processor to implement the method according to the first aspect of the present disclosure.

[0007] It is to be understood that the description of the background of the application included herein is not admitted to be prior art by its inclusion in this section. The present disclosure is directed to overcoming one or more of the problems discussed above. BRIEF DESCRIPTION OF DRAWINGS

[0008] The above and other features, aspects, and advantages of various implementations of the present disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which like reference numerals designate like elements in the drawings now described. In the drawings:

[0009] Figure 1 A block diagram illustrating an example environment in which implementations of the present disclosure can be implemented is shown;

[0010] Figure 2 A block diagram illustrating a process for managing sensitive information in visual data according to some implementations of the present disclosure is shown;

[0011] Figure 3A A block diagram illustrating a visual data authoring tool according to some implementations of the present disclosure is shown.

[0012] Figure 3B A block diagram illustrating an edit page for processing sensitive information according to some implementations of the present disclosure is shown;

[0013] Figure 4 A block diagram illustrating an edit tool for adjusting a sensitive range according to some implementations of the present disclosure is shown;

[0014] Figure 5 A block diagram illustrating a process for generating a blurred image according to some implementations of the present disclosure is shown;

[0015] Figure 6 A block diagram illustrating a process for determining a representative pixel based on pixels within an image tile according to some implementations of the present disclosure is shown;

[0016] Figure 7 A block diagram illustrating a process for determining a mask image from a blurred image according to some implementations of the present disclosure is shown;

[0017] Figure 8 A flow diagram illustrating a method for editing a view image according to some implementations of the present disclosure is shown;

[0018] Figure 9A flowchart illustrating a method for determining a mask image according to some implementations of the present disclosure is shown;

[0019] Figure 10 A flowchart illustrating a method for managing sensitive information in visual data according to some implementations of the present disclosure is shown;

[0020] Figure 11 A block diagram of an apparatus for managing sensitive information in visual data according to some implementations of the present disclosure is shown; and

[0021] Figure 12 A block diagram of an apparatus capable of implementing a number of implementations of the present disclosure is shown. DETAILED DESCRIPTION

[0022] Implementations of the present disclosure will be described below in detail with reference to the accompanying drawings. While certain implementations of the present disclosure are shown in the drawings, it is understood that the present disclosure can be embodied in various forms and should not be interpreted as being limited to the implementations set forth herein; rather, these implementations are provided so that the present disclosure can be more thoroughly and completely understood. It should be understood that the drawings and implementations of the present disclosure are only for illustrative purposes and should not be used to limit the scope of protection of the present disclosure.

[0023] In the description of implementations of the present disclosure, the term "comprising" and its conjugations should be understood to encompass the meanings of "including but not limited to", i.e., "comprising but not limited to". The term "based on" should be understood as "based at least in part on". The term "one implementation" or "the implementation" should be understood as "at least one implementation". The term "some implementations" should be understood as "at least some implementations". Other explicit and implicit definitions can also be included below. As used herein, the term "model" can represent the association between various data. For example, the above-mentioned association can be obtained based on various technical solutions that are currently known and / or will be developed in the future.

[0024] It can be understood that the data involved in the technical solutions of the present disclosure (including but not limited to the data itself, the acquisition or use of the data) should comply with the requirements of relevant laws and regulations and relevant provisions.

[0025] It can be understood that before using the technical solutions disclosed in the embodiments of the present disclosure, the type of personal information involved in the present disclosure, the scope of use, the scenario of use, etc. should be informed to the user and the authorization of the user should be obtained through appropriate means according to relevant laws and regulations.

[0026] For example, in response to receiving an active request of a user, a prompt information is sent to the user to explicitly prompt the user that the operation requested to be performed will require obtaining and using personal information of the user. Thus, the user can autonomously select whether to provide personal information to the software or hardware such as an electronic device, an application program, a server or a storage medium, etc. performing the operation of the technical solution of the present disclosure according to the prompt information.

[0027] As an optional but non-limiting implementation manner, in response to receiving an active request of a user, the manner of sending a prompt information to the user may, for example, be a pop-up window manner, and the prompt information may be presented in the form of text in the pop-up window. In addition, the pop-up window may also carry selection controls for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0028] It can be understood that the above notification and obtaining user authorization process is only illustrative and does not limit the implementation manner of the present disclosure, and other manners meeting relevant laws and regulations can also be applied to the implementation manner of the present disclosure.

[0029] Example Environment

[0030] In the context of the present disclosure, the visual data may, for example, include various types such as panoramic images and video sequences. For ease of description, the specific process of processing sensitive information in visual data will be described taking panoramic images as an example. First, the application environment of the present disclosure is described, Figure 1 A block diagram of an example environment 100 in which implementation manners of the present disclosure can be implemented is shown. As shown in Figure 1 As shown, in a panoramic image roaming application, visual data 110 (i.e., panoramic images) can be provided to provide an immersive roaming experience as the user's viewpoint moves.

[0031] It will be understood that there can be various sensitive information in the real world. For example, in an indoor environment, there can be sensitive information such as personal photos, portraits, or other text-related sensitive information; in an outdoor environment, there can be sensitive information such as human faces, vehicle license plates, etc. If the visual data 110 is directly generated based on the originally collected images, the visual data 110 can include sensitive information 112, which can expose sensitive data and cause potential risks. Thus, the sensitive information 112 should be removed when generating the visual data 110.

[0032] Typically, visual data 110 is generated based on multiple view images. For example, visual data 110 can be generated based on visual images 120 to 142 from multiple image acquisition devices. Specifically, view images 120 and 122 can represent a front view image and a rear view image associated with a specific viewpoint, respectively; view images 130 and 132 can represent a left view image and a right view image associated with a specific viewpoint, respectively; and view images 140 and 142 can represent a top view image and a bottom view image associated with a specific viewpoint, respectively.

[0033] Currently, technical solutions have been proposed for manually removing sensitive information from images using image editing tools. For example, a user can operate an image editing tool and blur sensitive information 122, such as faces, in view image 120 (e.g., add mosaic). However, this requires the user to open and edit each view image used as visual data 110 individually in the image editing tool. Furthermore, the user needs to open a panoramic image creation tool and import each view image with the sensitive information removed. If the processing effect is unsatisfactory, the user must return to the image editing tool, resulting in significant time and manpower costs. Therefore, a more efficient technical solution for processing sensitive information 112 in visual data 110 is desired.

[0034] Summary Process for Managing Sensitive Information

[0035] To address the shortcomings of the aforementioned technical solutions, a method for processing sensitive information 112 in visual data 110 is proposed. Specifically, multiple view images can be acquired to generate visual data 110 associated with a target viewpoint. For example... Figure 1 Regarding the visual data 110 shown, multiple view images 120, 122, 130, 132, 140, and 142 can be acquired.

[0036] Furthermore, the acquired multiple view images can be processed one by one to determine the sensitive range of the sensitive information 112 in each view image. See also Figure 2 To provide more details about the sensitive information, the Figure 2 A block diagram 200 illustrates a process for managing sensitive information in visual data according to some implementations of this disclosure. For example... Figure 2 As shown, the view image 120 used to generate visual data 110 includes sensitive information 122. At this point, the sensitive range within the view image 120 containing the sensitive information 122 can be determined. Further, a mask image 220 corresponding to the sensitive information 122 can be generated, and the mask image 220 can be used to update portions within the sensitive range in the view image 210, thereby obtaining an updated view image 210.

[0037] likeFigure 2 As shown, the mask image 220 is blurred in a mosaic manner with respect to the sensitive information 122. In this way, the sensitive information 122 can be removed from the original view image 120, and thus the updated view image 210 can be ensured to comply with various regulations associated with the sensitive information. Further, new visual data 230 associated with the target viewpoint can be generated based on the updated view image 210 and other view images in the plurality of view images except the view image 120. When using the panoramic image roaming tool, the visual data 230 with the sensitive information removed, i.e., the original sensitive information 122 replaced by the mask image 220, can be displayed.

[0038] With the example implementation of the present disclosure, a technical solution for processing sensitive information in visual data is provided, which can be invoked in the process of making visual data, and thus visual data without sensitive information is generated. In this way, the user does not have to switch between the image editing application and the panoramic image making tool, but can directly process the sensitive information in the panoramic image making tool. Thus, the making efficiency can be improved while ensuring the elimination of sensitive information.

[0039] Detailed Process for Managing Sensitive Information

[0040] In the following, more details about processing sensitive information will be described. According to one example implementation of the present disclosure, the method of the present disclosure can be performed in a panoramic image making tool. First, the process of obtaining a plurality of view images will be introduced. The plurality of view images herein refer to images to be used to synthesize a panoramic image, which can include at least one of the following: a left view image, a right view image, a front view image, a back view image, a top view image, and a bottom view image. The plurality of view images may, for example, be acquired by a panoramic image capturing device. Figure 1 The view images 120 and 122 (representing front and back views, respectively), the view images 130 and 132 (representing left and right views, respectively), and the view images 140 and 142 (representing top and bottom views, respectively) are shown. It will be understood that the view images 140 and 142 of the bottom view typically only include the ceiling and the floor, and thus these view images can be ignored in order to reduce the workload and time overhead.

[0041] According to one example implementation of the present disclosure, the plurality of view images acquired by a panoramic image capturing device can be directly loaded in the panoramic image making tool. It will be understood that the panoramic image capturing device can include a plurality of capturing devices, for example, six capturing devices for capturing six views of front, back, left, right, top, and bottom, respectively. At this time, the original images captured by the capturing devices can be directly loaded.

[0042] According to one example implementation of the present disclosure, a panoramic image can be first acquired (e.g., importing a previously made panoramic image in a panoramic image making tool). When sensitive information is found to be included in the panoramic image, a panoramic image algorithm can be invoked to divide the panoramic image into multiple view images. With example implementations of the present disclosure, both raw view images captured directly can be supported to be processed, and sensitive information can be removed after a panoramic image has been synthesized. In this way, multi-faceted support can be provided for the entire making process of a panoramic image.

[0043] In the following, the process of removing sensitive information from a view image will be described with view image 120 as an example only, other view images can be processed in a similar way. Figure 3A A block diagram 300A for a visual data making tool according to some implementations of the present disclosure is shown. As shown in 3, a visual data making tool can be used to generate visual data 110, when sensitive information 122 is found to be included in the visual data 110, an editing page of the present disclosure can be invoked for handling the sensitive information by pressing an edit button 310. Alternatively and / or additionally, the editing page can be invoked via a specific shortcut key.

[0044] Figure 3B A block diagram 300 of an editing page for handling sensitive information according to some implementations of the present disclosure is shown. As shown in 4, sensitive information can be handled in the editing page 320. The left side of the editing page 320 can include thumbnails of various view images associated with the visual data 110. For example, thumbnail 322 can be pressed to display view image 120, thumbnail 322 can be pressed to display another view image, and so on. Figure 3B

[0045] According to one example implementation of the present disclosure, if it is determined that sensitive information 122 is included in view image 120, a sensitive range 330 can be rendered around the sensitive information 122. The sensitive information 122 can be determined in a variety of ways, for example, the content of the image can be automatically recognized to provide a candidate of the sensitive information 122 to the user. The location of the sensitive information 122 can be automatically determined, and a sensitive range 330 that can cover the sensitive information 122 is automatically rendered around the location. For another example, an operation such as a click, a marquee, and the like on the sensitive information 122 from the user can be received, and in turn a corresponding sensitive range is rendered around the sensitive information 122. Specifically, the sensitive range 330 can be rendered around a 32x32 pixel range (or other range) of the click point, or the sensitive range 330 can be determined based on the size of the user's marquee region.

[0046] ​According to an exemplary implementation of this disclosure, adjustments to the sensitive range 330 by a user can be received, and the sensitive range 330 can be updated based on these adjustments. See below. Figure 4 More details regarding the adjustment of sensitive area 330 are described below. Figure 4 A block diagram 400 shows an editing tool for adjusting the sensitivity range 330 according to some implementations of this disclosure. For example... Figure 4 As shown, multiple adjustment tools can be presented on the editing page 320 to adjust the sensitive area 330 around the sensitive information 122. Specifically, multiple adjustment tools can be presented as a floating window after the user selects the sensitive area 330; alternatively and / or additionally, the adjustment tool can be displayed at the edge of the editing page 320.

[0047] like Figure 4 As shown, the size of the sensitive area 330 can be adjusted horizontally, vertically, and diagonally using the scaling tool 410; the position of the sensitive area 330 can be moved up, down, left, and right using the pan tool 420; the orientation of the sensitive area 330 can be rotated using the rotation tool 430; and the sensitive area 330 can be deleted from the view image 120 using the deletion tool 440. Alternatively and / or additionally, contour points 460 can be defined in the outline of the sensitive area 330, and the position of the contour points 460 can be changed by dragging or other operations. Using the exemplary implementation of this disclosure, a variety of adjustment methods provide rich support for adjusting the sensitive area 330, thereby allowing the sensitive area 330 to be adjusted in a more precise manner, so that the mask image 220 matches the edge of the sensitive area as closely as possible without obscuring too much area in the visual data.

[0048] It will be understood that although the foregoing has only described a specific example of defining and adjusting the sensitive range 330 using a rectangle as an example, according to one exemplary implementation of this disclosure, the sensitive range 330 may also be represented by other shapes. For example, other shapes may include, but are not limited to, rectangles, squares, circles, ellipses, polygons, and irregular shapes drawn from other line segments and / or curves. According to one exemplary implementation of this disclosure, various editing tools allow for the overall movement, rotation, and scaling of individual shapes, and the individual shape outlines can be adjusted individually to define the position of a given point.

[0049] According to one example implementation of the present disclosure, the blur degree 450 can indicate a degree of blurring processing for the sensitive information. For example, three blur degrees of high, medium, and low can be provided for user selection. Here, high can represent performing high degree of blurring processing, at which a mask image including larger mosaics can be used to replace the portion within the sensitive range 330; and low can represent performing low degree of blurring processing, at which a mask image including smaller mosaics can be used to replace the image within the sensitive range 330.

[0050] With the example implementation of the present disclosure, multiple blur degrees are provided to allow the user to select a suitable blur degree according to his / her own needs. For example, a high degree mode can be used to replace a large portrait, and a low degree mode can be used to replace sensitive information including smaller text, etc. In this way, the fusion degree of the mask image 220 and the surrounding original image can be taken into account while removing the sensitive information 122, thereby reducing the influence of the mosaic image on the overall browsing effect of the panoramic image.

[0051] According to one example implementation of the present disclosure, the mask image can be determined based on the blur degree selected by the user. Specifically, after the view image 120 is selected, multiple blurred images of the view image 120 can be respectively generated according to multiple predetermined blur degrees. Figure 5 A block diagram 500 of a process for generating a blurred image according to some implementations of the present disclosure is shown. As Figure 5 shown, the view image 120 can be divided into image blocks (e.g., image blocks 510 and 520) with different sizes according to different blur degrees, and then blurred images 512 and 522 with different blur degrees are respectively generated. The division of the view image 120 into N x N image blocks can be defined, and the division parameters of the high, medium, and low blur degrees can be represented as N 高 , N 中 , and N 低 , respectively. It can be defined that N 高 < N 中 < N 低 .

[0052] In the following, specific details of generating a blurred image are described with the low and high blur degrees as examples. As Figure 5 shown, the view image 120 can be divided based on smaller image blocks 510 (e.g., 8 x 8 pixels). For each image block obtained by the division, a representative pixel of the image block can be determined using the pixels within the image block. Further, based on the respective representative pixels of the multiple image blocks, a blurred image of the multiple blurred images matching a predetermined blur degree can be generated.

[0053] According to an exemplary implementation of this disclosure, representative pixels of image block 510 can be determined in various ways. For example, representative pixels can be determined based on pixels at predetermined positions within image block 510. Representative pixels can be determined using pixels at the center of image block 510, and alternatively and / or additionally, a pixel within image block 510 can be randomly selected as the representative pixel. Representative pixels can also be determined based on the average of multiple pixels within image block 510. For example, each pixel in image block 510 can be iterated over, each pixel can be summed, and the average of the sums can be used as the representative pixel.

[0054] Figure 6 A block diagram 600 illustrates a process for determining representative pixels based on pixels within image block 510, according to some implementations of this disclosure. For example... Figure 6 As shown, each pixel 610, ..., 620 in image block 510 can be scanned line by line, and the average of all pixels is used as the representative pixel of image block 510. According to an exemplary implementation of this disclosure, each pixel may include four RGBA channels, and the pixel data in each channel can be processed separately in a similar manner. For example, the variables red, green, blue, and alpha can be used to store the sum of the RGBA pixel data in each channel. Furthermore, the variables red, green, blue, and alpha can be divided by the number of pixels in image block 510 to determine the RGBA pixel data representing the pixel.

[0055] In this way, a pixel can represent the overall color trend within an image patch. For example... Figure 5 As shown, although the blurred image 512 is somewhat blurred, it can still represent the interior environment of the room; that is, objects such as walls, doors, and picture frames can still be roughly identified. Using the blurred image 512, the blurred image, determined based on the representative pixels of each image block, can represent the content of the view image 120, thereby allowing the color distribution of the generated mask image to match the color distribution within the sensitive range 330. In this way, the color distribution of the mask image can be kept as consistent as possible with the original view image 120, thus weakening the adverse effects of the mask image on the overall visual effect.

[0056] return Figure 5 For higher levels of blur, the view image 120 can be divided into a smaller number of image blocks according to the size of image block 520 (e.g., 16×16 pixels). In this case, each image block is larger, and the mosaic in the resulting blurred image 522 is larger, thereby providing a higher degree of blur.

[0057] It will be appreciated that, in determining the representative pixels based on the mean value of the pixels within the image block, a large amount of computation can be caused due to the need to traverse each pixel in the view image 120. At this point, a sub-process can be created in the main process for performing processing of sensitive information. In this way, the problem of lag of the main process caused by traversing the pixels to generate the blurred image can be reduced. Specifically, if a request for activating the method is detected in the visual data production tool, a sub-thread can be started in the main thread for performing processing of sensitive information, so as to generate a plurality of blurred images of the view image 120 by using the sub-thread. Further, if it is determined that the plurality of blurred images have been successfully generated, the main thread can be returned from the sub-thread.

[0058] According to one exemplary implementation of the present disclosure, in order to avoid the process of generating the blurred image affecting the drawing process of the user interface, the process of calculating the blurred image and drawing the user interface can be coordinated based on Web Worker and OffscreenCanvas technologies. It will be appreciated that the JS language usually adopts single-threaded execution when being executed, i.e. the single thread can only execute one task at the same time. When running in a synchronous execution manner, if a blockage occurs, the following task will not be executed. The Web Worker technology is proposed in HTML5, at which point the JS voice can allow multiple threads. For example, the main thread can start a sub-thread, and the sub-thread can be completely controlled by the main thread. The sub-thread cannot operate the Document Object Module (abbreviated as DOM), and only the main thread can operate the DOM.

[0059] According to one exemplary implementation of the present disclosure, the Web Worker can start a sub-thread on the basis of single-threaded execution, execute additional tasks without affecting the execution of the main thread, and return to the main thread after the sub-thread is executed, which does not affect the execution process of the main thread. At this point, the sub-thread can be used to execute tasks related to determining the blurred image and determining the mask image. In this way, the speed of determining the mask image can be improved without affecting the main thread to draw the user interface.

[0060] OffscreenCanvas is mainly used to improve the rendering performance and user experience of Canvas 2D / 3D drawing applications. OffscreenCanvas can be used in the following ways: for example, an OffscreenCanvas is created in a Worker thread and used for background rendering, and then the rendered buffer is returned to the main thread for display; for another example, the main thread can generate an OffscreenCanvas from a Canvas element in the current DOM tree, and send the OffscreenCanvas to the Worker thread for rendering, and the rendering result is directly submitted to the current window of the browser, which is equivalent to directly updating the content of the Canvas element in the Worker thread.

[0061] According to one example implementation of the present disclosure, since the main thread and the Worker thread are not in the same context, messages can be sent through postMessage and received through onmessage events (which can be through the two APIs of addEventListener or onmessage). Specifically, the main thread and the worker thread can be implemented based on the processes shown in Table 1 and Table 2 below, respectively.

[0062] Table 1: Example process of the main thread

[0063]

[0064] Table 2: Example process of the worker thread

[0065]

[0066] According to one example implementation of the present disclosure, for OffscreenCanvas, before obtaining the Canvas2D (2D canvas) context, the control of the canvas element in the main thread can be transferred to the worker thread through transferControlToOffscreen. The context of Canvas2D can be obtained in the worker thread using getContext, at which time the main thread can no longer operate Canvas2D.

[0067] According to one example implementation of the present disclosure, blurred images of different blur levels can be calculated in the worker thread. Alternatively and / or additionally, the process for adjusting the sensitive area 330 can be performed in the worker thread. With the example implementation of the present disclosure, the speed of determining blurred images can be improved without interfering with the rendering of the user interface. Further, when the user switches the viewpoint in the panoramic image, the display of the panoramic animation will be smoother.

[0068] According to one example implementation of the present disclosure, a blur image matching the user-selected blur level can be selected from a plurality of blur images. Figure 7 A block diagram 700 of a process for determining a mask image from a blur image is shown according to some implementations of the present disclosure. For example, if a user selects a low blur level, a blur image 512 can be selected as shown Figure 5 The blur image 512 is shown, and further, a region corresponding to the sensitive range 330 is selected in the blur image 512 in order to generate a mask image 220. As shown Figure 7 It is assumed that the location of the sensitive range 330 is (x0, y0, w, h), where x0, y0may represent the coordinates of the top-left corner of the sensitive range 330, and w, h can represent the width and height of the rectangular sensitive range 330. The portion at the location (x0, y0, w, h) in the blur image 512 can be taken as the mask image 220.

[0069] According to one example implementation of the present disclosure, when adjusting the size and location of the sensitive range 330, the mask image 220 should be adjusted accordingly. In other words, the corresponding mosaic pixels within the sensitive range 330 should be acquired in real time. The above process involves operations such as component re-rendering and canvas drawing, which can cause delays and lag during the process of using the visual data making tool.

[0070] In order to improve the response speed of the visual data making tool, when opening the editing tool, blur images of 3 blur levels can be determined for 4 view images of different directions (excluding the top view image and the bottom view image), and the generated plurality of blur images can be stored in the memory of the computing device. When detecting that the sensitive range 330 is adjusted, the currently selected blur level can be acquired, and the pixels at the matching location (x0, y0, w, h) in the corresponding blur image in the memory can be selected as the mask image 220 in order to replace the content of the sensitive range 330 in the view image 120. In this way, the mask image 220 can be determined at a faster speed, and in turn, the risk of potential delays can be reduced.

[0071] Figure 8 A flowchart of a method 800 for editing a view image is shown according to some implementations of the present disclosure. According to one example implementation of the present disclosure, the method 800 can be executed in a main thread. Specifically, at block 810, it can be detected that a user enters an editing page in the visual data making tool. At block 820, a plurality of view images associated with the visual data can be acquired. At block 830, a user-selected view image can be received. At block 832, a sub-thread can be started and a mask image can be determined in the sub-thread. It will be appreciated that the sub-thread can run in parallel with the main thread in order to reduce the impact on the user interface drawing operations of the main thread.

[0072] Further, at block 840, the mask image returned by the sub-thread can be received, and the sensitive region 330 in the view image 120 can be updated based on the mask image. At block 850, the mask image can be edited based on the user’s adjustment to the sensitive region 330. In other words, the portion of the blurred image corresponding to the sensitive region can be selected as the mask image based on the position and shape of the modified sensitive region.

[0073] At block 860, it can be determined whether all view images have been processed. If all selected view images have been successfully processed, the method 800 can proceed to block 870 to submit the processed view images, and further generate new visual data based on the processed view images. If there are still unprocessed view images, the method 800 can return to block 830 to receive the next view image selected. Each view image selected can be processed in a similar manner, and further generate new visual data based on the processed view images. With the exemplary implementation of the present disclosure, sensitive information can be directly processed in the visual data production tool, thereby improving the efficiency of visual data production.

[0074] Figure 9 A flowchart of a method 900 for determining a mask image according to some implementations of the present disclosure is shown. Specifically, at block 910, a canvas can be initialized based on a received view image, and at block 920, an image can be drawn. At block 930, an original view image can be obtained, and at block 940, a sub-thread can be launched based on the OffscreenCanvas and Web Worker techniques described above, and further determine blurred images corresponding to high, medium, and low levels, respectively. After the blurred images have been successfully determined, the main thread can be returned.

[0075] Further, at block 960, it can be determined whether all view images have been processed. If so, the method 900 can proceed to block 970 and end; otherwise, the method 900 can proceed to block 980 to process other view images. Further, the method 900 can return to block 920 to draw the current view image. With the exemplary implementation of the present disclosure, the multi-threading technique can be utilized to process the processes related to interface display with the main thread, and further process the processes associated with blurred image calculation with the sub-thread. In this way, multiple tasks can be processed in parallel, thereby reducing the lag of the user interface.

[0076] The process of processing sensitive information in a panoramic image in a multi-thread manner has been described above. Alternatively and / or additionally, the steps related to sensitive information processing can be performed in a single thread based on a requestIdleCallback technique during idle periods of the main thread. Specifically, the process of determining the blurred image of three blur levels of high, medium and low can be divided into four tasks. Further, the above-mentioned tasks can be performed in multiple idle periods of the main thread. In this way, although the phenomenon of lag cannot be completely eliminated, by making full use of the idle periods of the main thread, the process of processing sensitive information can be made to interfere as little as possible with the process of drawing the user interface related to the thread, thereby reducing the frequency of the phenomenon of lag.

[0077] Although the process of processing sensitive information in visual data has been described above with panoramic image as a specific example of visual data, according to one example implementation of the present disclosure, the visual data can also include a video sequence. At this time, the view image can be each video frame in the video sequence. Specifically, the sensitive object in one video frame can be marked based on the method described above. Further, the sensitive object appearing in the video sequence can be continuously tracked across multiple video frames based on object tracking technology. Continuing the example of the portrait of a person described above, in the case of identifying that one video frame includes the portrait of a person based on manual and / or automatic manner, the portrait of the person can be identified within multiple video frames before and after the video frame, and further the portrait of the person appearing in each video frame can be obscured by using a mosaic.

[0078] It will be understood that although the process of generating a mask image has been described above with a mosaic as an example of blurring processing, according to one example implementation of the present disclosure, the mask image can be generated based on other ways. For example, the mask image can be generated based on Gaussian blur, noise blur, obscuring blur, motion blur and other ways.

[0079] With the example implementation of the present disclosure, the function of processing sensitive information can be inherited in a variety of visual data production applications, and in turn the visual data can be produced with higher efficiency while removing sensitive information from the visual data can be ensured.

[0080] Example Process

[0081] Figure 10A flowchart of a method 1000 for processing sensitive information in visual data according to some implementations of the present disclosure is shown. Specifically, at block 1010, a plurality of view images used to generate visual data associated with a target viewpoint are obtained. At block 1020, for a view image of the plurality of view images, a sensitive range of sensitive information in the view image is determined. At block 1030, a portion within the sensitive range in the view image is updated with a mask image. At block 1040, the visual data associated with the target viewpoint is generated based on the updated view image and other view images of the plurality of view images.

[0082] According to one example implementation of the present disclosure, determining the sensitive range comprises: in response to determining the sensitive information in the view image, presenting the sensitive range at a periphery of the sensitive information; and in response to receiving an adjustment for the sensitive range, updating the sensitive range.

[0083] According to one example implementation of the present disclosure, the adjustment comprises at least one of: a position change, an orientation change, a scaling, a deletion, an addition, a change of contour points of a region, for the sensitive range.

[0084] According to one example implementation of the present disclosure, further comprising: receiving a blur degree for blurring the sensitive information; and determining the mask image based on the blur degree.

[0085] According to one example implementation of the present disclosure, determining the mask image based on the blur degree comprises: generating a plurality of blurred images of the view image respectively based on a plurality of predetermined blur degrees; selecting a blurred image matching the blur degree from the plurality of blurred images; and selecting a region corresponding to the sensitive range in the blurred image so as to generate the mask image.

[0086] According to one example implementation of the present disclosure, generating the plurality of blurred images respectively comprises: for a predetermined blur degree of the plurality of predetermined blur degrees, dividing the view image into a plurality of image blocks based on the predetermined blur degree; for an image block of the plurality of image blocks, determining a representative pixel of the image block based on pixels within the image block; and generating a blurred image of the plurality of blurred images matching the predetermined blur degree based on the respective representative pixels of the plurality of image blocks.

[0087] According to one example implementation of the present disclosure, determining the representative pixel comprises at least one of: determining the representative pixel based on a pixel at a predetermined position within the image block; and determining the representative pixel based on a plurality of pixels within the image block.

[0088] According to one example implementation of the disclosure, generating the plurality of blurred images of the view image includes: in response to detecting a request for activating the method in the visual data production tool, starting a sub-thread in a main thread of the method to generate the plurality of blurred images of the view image with the sub-thread; and in response to determining that the plurality of blurred images have been successfully generated, returning the main thread from the sub-thread.

[0089] According to one example implementation of the disclosure, the visual data is a panorama image, the plurality of view images includes at least one of the following: a left view image, a right view image, a front view image, a back view image, a top view image, a bottom view image of the panorama image taken from the target viewpoint, and the obtaining the plurality of view images of the panorama image includes at least one of the following: splitting the panorama image to obtain the plurality of view images; and receiving the plurality of view images captured by a panorama image taking device.

[0090] According to one example implementation of the disclosure, the visual data includes a video sequence, and the plurality of view images includes a plurality of consecutive video frames in the video sequence.

[0091] Example Devices and Equipment

[0092] Figure 11 A block diagram of an apparatus 1100 for processing sensitive information in visual data according to some implementations of the disclosure is shown. The apparatus 1100 includes: an obtaining module 1110 configured to obtain a plurality of view images for generating visual data associated with a target viewpoint; a determining module 1120 configured to determine, for a view image of the plurality of view images, a sensitive range of sensitive information in the view image; an updating module 1130 configured to update a portion within the sensitive range in the view image with a mask image; and a generating module 1140 configured to generate the visual data associated with the target viewpoint based on the updated view image and other view images of the plurality of view images except the view image.

[0093] According to one example implementation of the disclosure, the determining module 1120 includes: a range presenting module configured to present the sensitive range around the sensitive information in response to determining the sensitive information in the view image; and an adjusting module configured to update the sensitive range in response to receiving an adjustment for the sensitive range.

[0094] According to one example implementation of the disclosure, the adjustment includes at least one of the following: a position change, an orientation change, a zoom, a deletion, an addition, a change of contour points of an area for the sensitive range.

[0095] According to one example implementation of the present disclosure, the method further includes receiving a blur degree for blurring the sensitive information; and determining the mask image based on the blur degree.

[0096] According to one example implementation of the present disclosure, the mask image determining module includes: an image generating module configured to generate a plurality of blurred images of the view image based on a plurality of predetermined blur degrees; a matching module configured to select a blurred image matching the blur degree from the plurality of blurred images; and a selecting module configured to select a region corresponding to the sensitive range in the blurred image so as to generate the mask image.

[0097] According to one example implementation of the present disclosure, the image generating module includes: a dividing module configured to divide, for a predetermined blur degree in the plurality of predetermined blur degrees, the view image into a plurality of image blocks based on the predetermined blur degree; a representative pixel determining module configured to determine, for an image block in the plurality of image blocks, a representative pixel of the image block based on pixels within the image block; and a blurred image generating module configured to generate, based on respective representative pixels of the plurality of image blocks, a blurred image matching the predetermined blur degree in the plurality of blurred images.

[0098] According to one example implementation of the present disclosure, the representative pixel determining module includes at least one of: a first determining module configured to determine the representative pixel based on a pixel at a predetermined position within the image block; and a second determining module configured to determine the representative pixel based on a plurality of pixels within the image block.

[0099] According to one example implementation of the present disclosure, the image generating module includes: a starting module configured to start, in response to detecting a request for activating the device in a visual data making tool, a sub-thread in a main thread of the device so as to generate, by the sub-thread, a plurality of blurred images of the view image; and a returning module configured to return, in response to determining that the plurality of blurred images have been successfully generated, the main thread from the sub-thread.

[0100] According to one example implementation of the present disclosure, the visual data is a panorama image, the plurality of view images includes at least one of: a left view image, a right view image, a front view image, a back view image, a top view image, and a bottom view image of the panorama image taken from a target viewpoint, and the obtaining module includes at least one of: a splitting module configured to split the panorama image to obtain the plurality of view images; and a receiving module configured to receive the plurality of view images collected by a panorama image shooting device.

[0101] According to one example implementation of the present disclosure, the visual data comprises a video sequence and the plurality of view images comprises a plurality of consecutive video frames in the video sequence.

[0102] Figure 12 A block diagram of a device 1200 capable of implementing the various implementations of the present disclosure is shown. It should be understood that Figure 12 The computing device 1200 shown is merely exemplary and should not be construed as limiting the functionality and scope of the implementations described herein. Figure 12 The computing device 1200 shown can be used to implement the methods described above.

[0103] As Figure 12 The computing device 1200 is in the form factor of a general- purpose computing device, as shown. Components of the computing device 1200 can include, but are not limited to, one or more processors or processing units 1210, a memory 1220, a storage device 1230, one or more communication units 1240, one or more input devices 1250, and one or more output devices 1260. The processing units 1210 can be actual or virtual processors and are capable of executing various processing in accordance with programs stored in the memory 1220. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capabilities of the computing device 1200.

[0104] The computing device 1200 typically includes a plurality of computer storage media. Such media can be any available media that is accessible by the computing device 1200 and includes both volatile and non-volatile media, removable and non-removable media. The memory 1220 can be volatile memory (such as a register, cache, random access memory (RAM)), non-volatile memory (such as read only memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory), or some combination thereof. The storage device 1230 can be a removable or non-removable media and can include machine-readable media, such as a flash drive, a disk drive, or any other media that can be used to store information and / or data (e.g., training data for training) and can be accessed by the computing device 1200.

[0105] The computing device 1200 can further include additional removable / non-removable, volatile / non-volatile storage media. For example, computer storage media can include, but is not limited to, Blu-ray discs, DVDs, CD-ROMs, flash memory drives, magnetic and / or optical disks, and / or other media that can be used to store information and / or data (e.g., training data for training) and accessed by the computing device 1200. Figure 12As shown in FIG. 12, a disk drive 1215 or other computer readable media storage drive can be provided for reading from or writing to a non-removable, nonvolatile magnetic media (e.g., a "hard drive"). A disk drive 1220 or other computer readable media storage drive can also be provided for reading from or writing to a removable, nonvolatile media (e.g., a floppy disk, a flash drive, a CD-ROM, a DVD, or other medium). In these instances, each drive can be connected to the bus by one or more data media interfaces. The memory 1220 can include a computer program product 1225 having one or more program modules configured to carry out the various methods or actions of the various implementations of the present disclosure.

[0106] The communication unit 1240 enables communications with other computing devices over a communication medium. Additionally, the functionality of the components of the computing device 1200 can be implemented in a single computing cluster or multiple computer machines that are capable of communicating over a communication connection. Thus, the computing device 1200 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network nodes in the networking environment.

[0107] The input device 1250 can be one or more input devices, such as a mouse, a keyboard, a trackball, etc. The output device 1260 can be one or more output devices, such as a display, a speaker, a printer, etc. The computing device 1200 can also communicate with one or more external devices (not shown) such as a storage device, a display device, etc. through the communication unit 1240, as needed, a device that enables a user to interact with the computing device 1200, or any device (e.g., a network card, a modem, etc.) that enables the computing device 1200 to communicate with one or more other computing devices. Such communication can be carried out via an input / output (I / O) interface (not shown).

[0108] According to example implementations of the present disclosure, a computer readable storage medium is provided having computer executable instructions stored thereon, where the computer executable instructions are executed by a processor to implement the methods described above. According to example implementations of the present disclosure, a computer program product is also provided that is tangibly stored on a non-transitory computer readable medium and includes computer executable instructions, where the computer executable instructions are executed by a processor to implement the methods described above. According to example implementations of the present disclosure, a computer program is provided having computer executable instructions that, when executed by a processor, implement the methods described above.

[0109] Various aspects of the disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, and computer program products according to this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0110] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can include a non-transitory computer readable storage medium that can be a computer- readable storage medium having no data storage viruses or other code or instructions implementing a functionally equivalent to that of the software manual reproduction process. The instructions can form an interface to other code or programs that or adapted at one time to implement specific processes or can form a component of another program, which can implement specific processes as conditions require.

[0111] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0112] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0113] The implementations of the disclosure have been described above with the intent to be illustrative rather than limiting. Although being described in the context of particular implementations, those skilled in the art will recognize that the implementations are not limited thereto and will apply to any implementation that falls within the scope of the implementations. Changes in form and detail can be made without departing from the spirit of the implementations. The implementation described herein is to be considered in a descriptive sense only and not limiting. Therefore, various modifications can be made to the implementations described and illustrated herein, consistent with the scope of the implementations as defined by the following claims.

Claims

1. A method for processing sensitive information in visual data, comprising: presenting, in a panorama authoring tool, visual data associated with a target viewpoint, the visual data being a panorama image generated based on a plurality of view images associated with the target viewpoint; in response to receiving an interaction request for editing the visual data, presenting, in the panorama authoring tool, an edit page including the plurality of view images of the visual data associated with the target viewpoint; determining, for a view image of the plurality of view images, a sensitive range of sensitive information in the view image; updating a portion within the sensitive range in the view image with a mask image; and updating, based on the updated view image and other view images of the plurality of view images, the panorama image associated with the target viewpoint.

2. The method of claim 1, wherein determining the sensitive range comprises: in response to determining the sensitive information in the view image, presenting the sensitive range around a periphery of the sensitive information; and in response to receiving an adjustment for the sensitive range, updating the sensitive range.

3. The method of claim 2, wherein the adjustment comprises at least one of a position change, an orientation change, a zoom, a deletion, an addition, a change of a contour point of the sensitive range, for the sensitive range.

4. The method of claim 3, further comprising: receiving a blur degree for blurring the sensitive information; and determining the mask image based on the blur degree.

5. The method of claim 4, wherein determining the mask image based on the blur degree comprises: generating a plurality of blurred images of the view image respectively based on a plurality of predetermined blur degrees; selecting a blurred image matching the blur degree from the plurality of blurred images; and selecting, in the blurred image, an area corresponding to the sensitive range so as to generate the mask image. for a predetermined blur degree of the plurality of predetermined blur degrees, dividing the view image into a plurality of image blocks based on the predetermined blur degree; for an image block of the plurality of image blocks, determining a representative pixel of the image block based on pixels within the image block; and generating, based on respective representative pixels of the plurality of image blocks, a blurred image matching the predetermined blur degree of the plurality of blurred images.

6. The method of claim 5, wherein generating the plurality of blurred images, respectively, comprises:

7. The method of claim 6, wherein determining the representative pixel comprises at least one of: determining the representative pixel based on a pixel at a predetermined position within the image block; and determining the representative pixel based on a plurality of pixels within the image block.

8. The method of claim 5, wherein generating the plurality of blurred images respectively comprises: in response to detecting, in the panorama authoring tool, a request for activating the method, starting a sub-thread in a main thread executing the method so as to generate the plurality of blurred images of the view image with the sub-thread; and ​ ​ ​ ​ ​ ​ returning, from the sub-thread, the main thread in response to determining that the plurality of blurred images have been successfully generated.

9. The method of claim 1, wherein the visual data is a panoramic image, and the plurality of view images comprises at least one of the following: a left view image, a right view image, a front view image, a back view image, a top view image, a bottom view image, of the panoramic image taken from the target viewpoint, and the acquiring the plurality of view images of the panoramic image comprises at least one of the following: splitting the panoramic image to acquire the plurality of view images; and receiving a plurality of view images captured by a panoramic image taking device.

10. The method of claim 1, wherein the visual data comprises a video sequence, and the plurality of view images comprises a plurality of consecutive video frames in the video sequence.

11. An apparatus for processing sensitive information in visual data, comprising: a presenting module configured for presenting, in a panoramic image authoring tool, visual data associated with a target viewpoint, the visual data being a panoramic image generated based on a plurality of view images associated with the target viewpoint; the presenting module is further configured for presenting, in the panoramic image authoring tool, an editing page comprising the plurality of view images of the visual data associated with the target viewpoint in response to receiving an interaction request for editing the visual data; a determining module configured for determining, for a view image of the plurality of view images, a sensitive range of sensitive information in the view image; an updating module configured for updating a portion within the sensitive range in the view image with a mask image; and a generating module configured for updating the visual data associated with the target viewpoint based on the updated view image and other view images of the plurality of view images other than the view image.

12. The apparatus of claim 11, wherein the determining module comprises: a range presenting module configured for presenting the sensitive range around a perimeter of the sensitive information in response to determining the sensitive information in the view image; and an adjusting module configured for updating the sensitive range in response to receiving an adjustment for the sensitive range.

13. The apparatus of claim 12, wherein the adjustment comprises at least one of the following: a position change, an orientation change, a zoom, a deletion, an addition, a change of a contour point of the sensitive range for the sensitive range.

14. The apparatus of claim 13, further comprising: a receiving module configured for receiving a blur degree for blurring the sensitive information; and a mask image determining module configured for determining the mask image based on the blur degree.

15. The apparatus of claim 14, wherein the mask image determining module comprises: an image generating module configured for generating a plurality of blurred images of the view image respectively based on a plurality of predetermined blur degrees. a matching module configured to select a blurred image matching the blur degree from the plurality of blurred images; and a selecting module configured to select a region corresponding to the sensitive range in the blurred image so as to generate the mask image.

16. The apparatus of claim 15, wherein the image generating module comprises: a dividing module configured to divide, for a predetermined blur degree of the plurality of predetermined blur degrees, the view image into a plurality of image blocks based on the predetermined blur degree; a representative pixel determining module configured to determine, for an image block of the plurality of image blocks, a representative pixel of the image block based on at least one pixel within the image block; and a blurred image generating module configured to generate, based on the respective representative pixels of the plurality of image blocks, a blurred image matching the predetermined blur degree of the plurality of blurred images.

17. The apparatus of claim 15, wherein the image generating module comprises: a starting module configured to start, in response to detecting a request for activating the apparatus in a panorama image making tool, a sub-thread in a main thread of the apparatus so as to generate the plurality of blurred images of the view image by the sub-thread; and a returning module configured to return, in response to determining that the plurality of blurred images have been successfully generated, the main thread from the sub-thread.

18. The apparatus of claim 11, wherein the visual data is a panorama image, the plurality of view images comprises at least one of the following: a left view image, a right view image, a front view image, a back view image, a top view image, a bottom view image of the panorama image taken from the target viewpoint, and the acquiring module comprises at least one of the following: a splitting module configured to split the panorama image to acquire the plurality of view images; and a receiving module configured to receive the plurality of view images captured by a panorama image capturing device.

19. An electronic device, comprising: at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions which when executed by the at least one processing unit cause the electronic device to perform the method according to any one of claims 1-10.

20. A computer readable storage medium having stored thereon a computer program which, when executed by a processor, causes the processor to implement the method according to any one of claims 1-10.

Images