Privacy-protected multi-path Street View photo stitching

Abstract

receiving a plurality of images of a location from a user device, the plurality of images including images of the location at various times; identifying an object in one or more of the plurality of images, the object corresponding to an unsuitable condition for the database; determining a score for one or more of the plurality of images based at least in part on the identified object; determining a base image from the one or more images of the plurality of images; and generating a set of replacement images of the location based at least in part on the determined scores for each of the one or more images of the plurality of images, thereby generating a controllable panoramic image while eliminating unsuitable dynamic elements.

Description

Technical Field

[0001] The present disclosure generally relates to image recognition. In particular, the present disclosure relates to combining data from multiple paths of a user device to generate a controllable panoramic image while excluding inappropriate dynamic elements.

Background Art

[0002] Virtual reality (VR) is a pseudo-experience that may be similar to or completely different from the real world. Applications of virtual reality include entertainment (such as video games), education (such as medical and military training), business (such as virtual meetings), etc. Other VR-style technologies include augmented reality (AR) and mixed reality (MR), which are sometimes referred to as extended reality (XR). A 360-degree photo is a controllable panoramic image that surrounds the original location where it was taken. A 360-degree photo simulates looking around left, right, up, and down as desired, and sometimes zooming, from the perspective of the photographer.

[0003] Computer vision is an interdisciplinary scientific field that deals with how a computer can obtain a high level of understanding from digital images and videos. From an engineering perspective, computer vision aims to understand and automate tasks that the human visual system can perform. Tasks in computer vision include the acquisition, processing, analysis, understanding of digital images, and methods for extracting high-dimensional data from the real world. Sub-domains of computer vision include scene reconstruction, object detection, event detection, video tracking, object recognition, 3D pose estimation, learning, indexing, motion estimation, 3D scene modeling, image restoration, etc.

Summary of the Invention

[0004] The following is a summary to provide a basic understanding of one or more embodiments of the present disclosure. This summary is not intended to identify any important parts or elements, or to define the scope of any particular embodiment or claim. One of its purposes is to present the concepts in a simplified form as a preliminary step to the more detailed descriptions that will be presented later. In one or more embodiments described herein, devices, systems, computer implementations, apparatuses, or computer program products, or combinations thereof, enable the combination of data from multiple paths of a user device to generate a controllable panoramic image while removing undesirable dynamic elements.

[0005] Aspects of the present invention disclose a method, system, and computer-readable medium relating to receiving a plurality of images of a location from a user device, the plurality of images including images of the location at various times; identifying an object in one or more of the plurality of images, the object corresponding to conditions unsuitable for a database; determining a score for one or more of the plurality of images at least in part on the identified object; determining a base image from one or more of the plurality of images; and generating a set of replacement images of the location at least in part on the determined score of each of the one or more of the plurality of images.

[0006] The above and other purposes, features, and advantages of this disclosure will become more apparent through a more detailed description of some embodiments of this disclosure in the accompanying drawings. [Brief explanation of the drawing]

[0007] [Figure 1] This is a schematic diagram of a computing environment according to one embodiment of the present invention. [Figure 2] This flowchart shows the operation sequence according to one embodiment of the present invention. [Figure 3] This shows a cloud computing environment using one embodiment of the present invention. [Figure 4] This figure shows an abstraction model layer according to one embodiment of the present invention. [Modes for carrying out the invention]

[0008] Several embodiments will be described in more detail with reference to the accompanying drawings illustrating embodiments of the present disclosure. However, the present disclosure can be implemented in various ways and should therefore not be construed as being limited to the embodiments disclosed herein.

[0009] Currently, various internet services allow users to view images of houses and businesses from the street. Generally, these images, captured by 360-degree cameras mounted on vehicles, are useful for a variety of tasks. Embodiments of the present invention recognize that these images may unduly infringe upon the privacy of individuals who happen to be present when the vehicle passes by. Conventional methods have made efforts to obscure individuals' faces and license plates. However, embodiments of the present invention recognize that there is a challenge in preventing the identification of individuals by obscuring faces and license plates, as individuals and their possessions may be identified by other means. Additional embodiments recognize that certain objects are unsuitable for a user's database for various reasons. For example, individuals or vehicles in an image may obscure the intended subject of the image (e.g., a storefront). In another example, an image may be captured at the moment a trash can overflows onto a sidewalk, and this unpleasant image may unfairly prejudice visitors against a place or business. Also, images may be captured that include billboard advertisements containing protected content (trademarks, copyrighted content, etc.) that is unsuitable for the database due to internal policies regarding the intellectual property of other companies.

[0010] The disclosed embodiments of the present invention enable the combination of data from multiple paths of a user device to generate a controllable panoramic image while removing unsuitable dynamic elements (e.g., objects corresponding to unsuitable conditions) from the database. Furthermore, the disclosed embodiments of the present invention utilize the temporal differences between multiple images of a given location to remove unsuitable dynamic elements. In addition, the disclosed embodiments automatically reduce the memory usage of the database by removing images containing unsuitable dynamic elements.

[0011] In one embodiment, one or more components of the system may employ hardware, software, or both to solve problems that are inherently highly technical (e.g., policy violation searches, identifying objects in one or more images out of several, where the objects correspond to conditions unsuitable for the database, etc.). These solutions are not abstract and cannot be performed as a series of mental acts by humans due to the processing power required to facilitate, for example, removing base images containing objects corresponding to unsuitable conditions from the database. Furthermore, some of the processing to be performed may be carried out by a dedicated computer for performing a defined task related to mapping a set of weighted labels to a set of replacement images, where the weighted labels correspond to objects in segments of one or more images out of several. For example, the dedicated computer may be employed to perform a task related to identifying objects in one or more images out of several, where the objects correspond to conditions unsuitable for the database, etc.

[0012] The embodiments of the present invention can be implemented in various forms, and details of exemplary implementations will be described later with reference to the figures.

[0013] In one embodiment, a system performing an inappropriate dynamic element removal method maps a weighted set of labels "L" to a set of one or more replacement images. This method generates a set of weighted labels by assigning values ​​(e.g., from 0.0 to 1.0, where 0.0 is the lowest and 1.0 is the highest) to the label "l" which indicates how undesirable an object in an image is to the user (i.e., assigning values ​​that indicate the unsuitability of an object in an image to the intended purpose of the database). The labels correspond to the user's unsuitability for the database. For example, unsuitability may be related to objects that cause problems including, but are not limited to, conflicts of interest, invasions of privacy, or offensive content. Conflicts of interest may include the use of images that contain the intellectual property of others or images that violate company policies. Invasions of privacy may include images that disclose personally identifiable information (PII) or other personally identifiable data elements. Offensive content may include the use of images that contain violent, threatening, fraudulent, or obscene content.

[0014] In embodiments, the method maps labels to a set of one or more replacement images. For example, the method can utilize a map function, which returns a map object (e.g., an iterator) of the results after applying a given function (e.g., a function that takes each element of a given iterable) to each item of a given iterable (e.g., a list, tuple, etc.). In this example, the method utilizes "map C(l,r)" where "C" is an object containing a returned list of results after applying a given function "l", "l" represents weighted labels, and "r" represents the iterable for mapping (e.g., replacement images of a set of replacement images). Furthermore, the method can use "C" to create lists, sets, or methods thereof, and the method can be organized by type (e.g., image segments, objects, pixel ranges, etc.).

[0015] In one embodiment, the method receives multiple geolocation images from a user device. The method receives multiple geolocation images from the user's imaging device via the Transmission Control Protocol (TCP) and the Internet Protocol (IP). The images may be panoramic street-level images of a building taken at various times while passing through the building multiple times. In this embodiment, the method uses the multiple images to generate a set of replacement images. Furthermore, the image and location data are provided by the imaging device and used in a method disclosed with the user's consent. In an alternative embodiment, the method retrieves multiple geolocation images from remote storage. The method receives a notification from the imaging device that includes a storage location indicating that the images are retrievalable from remote storage. In this alternative embodiment, the method uses the storage location in the notification to retrieve the images from remote storage.

[0016] This method trains a machine learning model, such as a recurrent neural network (RNN), support vector machine (SVM), or other classification machine learning model architecture, to identify objects corresponding to conditions unsuitable for the intended purpose of the database (i.e., to identify undesirable dynamic elements). The method trains the model to identify objects associated with weighted labels corresponding to unsuitable conditions. In one embodiment, the method labels a set of images containing features corresponding to objects of a class corresponding to an unsuitable condition and provides the set of labeled images for training the machine learning model. In one embodiment, the method reserves a portion of the set of labeled image data for use as test data to validate the trained machine learning model. The trained machine learning model enables the detection of objects corresponding to unsuitable conditions using new, unlabeled image data from the user's imaging device. The trained model provides an output showing the detection and classification of one or more objects corresponding to unsuitable conditions by the trained model from the new input image data. The classification of one or more objects corresponding to unsuitable conditions is determined by the method according to the output of the trained model.

[0017] In this embodiment, the method inputs one or more images (e.g., (p) is one of the multiple images) from a user's imaging device (e.g., a set of images "P") into a trained model and determines whether one or more of the multiple images contain an object corresponding to an inappropriate condition (e.g., a label, a class, etc.). In this embodiment, the method assigns a label to the image segment (e.g., a defined region) that contains the object corresponding to the inappropriate condition.

[0018] In one embodiment, the method determines the score of image (p) from a set of images from a user's imaging device. The method assigns a score to an image using the count of members of a set of labels "L" corresponding to identified objects in the image, and the weight of each member. For example, the method determines the score using Σ(L(i)), where "i" is the label "l" for each identified object in the image. In this example, the method can be used to determine the score associated with an image segment (e.g., a sub-image, a defined region, a pixel range, etc.). The method can also determine the score for each image (p) in a set of images "P" received from the user's imaging device.

[0019] In one embodiment, the method determines a base image (p') of multiple images from the user's imaging device. The method identifies the image with the lowest determined score for each of the multiple images. In this embodiment, the method selects the image with the lowest determined score from the multiple images and designates that image as the base image. Alternatively, the method can select a first image received from the user's imaging device as the base image. In another embodiment, the method selects a user-selected image provided by the user's device as the base image.

[0020] In one embodiment, the method generates a set of replacement images for a base image. When images are received by the method from a user's imaging device, the method determines a score for each of the received images using the method described above. In this embodiment, the method compares the determined score of the base image with the determined score of the received image to determine whether the determined score of the received image is lower than or equal to the determined score of the base image. In one scenario, if the method determines that the determined score of the received image is less than or equal to the determined score of the base image, the method adds the received image to the replacement set for the base image. In another scenario, if the method determines that the determined score of the received image is greater than the determined score of the base image, the method does not add the received image to the replacement set for the base image. Alternatively, the method may discard the received image from the database on the grounds that it is unsuitable for use.

[0021] In another embodiment, the method generates a set of replacement subimages for a segment of a base image. The method generates a set of pairs "M" containing subimages for scoring pairs of a labeled segment (p') of a base image and each received image (p) of a set of images (e.g., a set of images "p"). In this embodiment, the method utilizes a set of formulas to input "M", which can be used to replace the labeled segment of the base image (p') with a subimage of the image (p). For example, if the method determines that (p'=p), the method adds the labeled segment of the base image (p') to "M", which has a score corresponding to the labeled segment. If the method determines that "p'≠p" (e.g., the first formula), the method evaluates the second formula in the set of formulas. In this example, if the method determines that the subimage (p) of the image is unlabeled and corresponds to a labeled segment of the base image (p'), the method adds the subimage (p) of the image to "M" with a score of zero (0.0). If the second expression is determined to evaluate to false, the method evaluates the third expression in the set of expressions. In this example, if the method determines that the sum of the labels of the sub-images of image (p) (e.g., the sub-image score) corresponds to a labeled segment of the base image (p'), and the sum of the labels of (p) is less than the score of the labeled segment of the base image (p') (e.g., the segment score), the method adds the sub-images of image (p) to "M" with the sum of the sub-image labels as the score.

[0022] In one embodiment, the method replaces a base image with a replacement image from a set of generated replacement images. The method identifies images from the set of generated replacement images that have a lower score than the base image. In this embodiment, if the method does not identify images from the set of generated replacement images that have a lower score than the base image, the method masks the object in the base image corresponding to the unsuitable condition. For example, the method adds an overlay layer to the base image containing the object corresponding to the unsuitable condition. In this example, the overlay layer includes an image that covers the object and blocks it from the viewer's view.

[0023] In another embodiment, the method replaces a segment of a base image with a subimage from a set of replacement subimages. The method can rank the subimages in the set of replacement subimages by score (e.g., cumulative score of an object) to identify the subimage in the set of replacement subimages that has the lowest score corresponding to the segment of the base image. In this embodiment, the method can replace the segment of the base image with the identified subimage if the score of the subimage is lower than the score of the label of the segment of the base image. In one scenario, if the method determines that the expression in the set of expressions evaluates to false, the method replaces the labeled segment (p') of the base image with the corresponding subimage of "M" that has the lowest score lower than the score of the labeled segment. In another scenario, if the method determines that the lowest-scoring subimage of "M" corresponding to the labeled segment of the base image (p') is not lower than the score of the labeled segment, the method masks the segment of the base image (p'). Masking includes, but is not limited to, mosaicing an image to cover the segment of the base image or adding an image to a layer that covers the segment.

[0024] In yet another embodiment, the method replaces segments of the base image using the labels of the segments of the base image and the set of generated weighted labels. The method determines that one or more labeled segments are keys of the set of generated weighted labels. In an embodiment, the method replaces the labels of one or more labeled segments identified as keys with replacement sub-images "r" of the set of replacement images (i.e., if the labeled segment of the base image (p') is the key of "C", the labeled segment of the base image (p') is replaced with replacement "r").

[0025] In one embodiment, the method returns a base image that includes replacement images. This method provides a new or modified base image that no longer includes objects that correspond to inappropriate conditions for the database or where the objects of the modified base image are masked. In an embodiment, the method can remove the replaced base image or the labeled segments of the base image from the database, thereby causing the database to conform to the user's policy or increasing the available memory resources of the database for additional suitable replacement images, or both. Further, the method provides a base image suitable for generating a panoramic street-level image that protects the privacy of pedestrians and prevents infringement of property rights.

[0026] Figure 1 is a schematic diagram of exemplary network resources related to the implementation of the disclosed invention. The present invention can be implemented in any processor of the disclosed elements that processes an instruction stream. As shown in the figure, the network-connected client device 110 wirelessly connects to the server subsystem 102. The client device 104 wirelessly connects to the server subsystem 102 via the network 114. The client devices 104 and 110 include an application program (not shown) and sufficient computing resources (processor, memory, network communication hardware) to execute the program.

[0027] As shown in FIG. 1, the server subsystem 102 includes a server computer 150. FIG. 1 shows a block diagram of components of the server computer 150 within a networked computer system 1000, according to an embodiment of the invention. It should be understood that FIG. 1 provides only an illustration of one embodiment and does not imply any limitation with respect to the environments in which different embodiments may be implemented. Without departing from the scope of the invention as recited in the claims, many modifications to the illustrated environments may be made by those skilled in the art. [[ID=##]] [[ID=##]]

[0028] [[ID=##]] The present invention may include various accessible data sources, such as client devices 104 and 110 and memory 158, which may contain personal information, content, or information that the user wishes not to be processed. Personal information includes personally identifiable information and highly sensitive personal information, as well as user information such as tracking information and geolocation information. Processing refers to any operation or set of operations performed on personal information, whether automated or not, including collection, recording, organizing, structuring, storing, adapting, altering, retrieving, consulting, using, disclosing by transmission, disseminating or otherwise making available, combining, restricting, erasing, or destroying. Program 175 enables authorized and secure processing of personal information. Program 175 provides informed consent along with notice regarding the collection of personal information, allowing the user to opt in or opt out of the processing of personal information. Consent can take several forms. Opt-in consent may require the user to take proactive action before personal information is processed. Alternatively, opt-out consent may require the user to take proactive action to prevent the processing of personal information before it is processed. Program 175 provides information about personal information and the nature of its processing (type, scope, purpose, duration, etc.). Program 175 provides users with a copy of their stored personal information. Program 175 allows for the correction or completion of inaccurate or incomplete personal information. Program 175 allows for the immediate deletion of personal information.

[0029] The server computer 150 may include a processor 154, memory 158, persistent storage 170, a communication unit 152, an input / output (I / O) interface 156, and a communication fabric 140. The communication fabric 140 provides communication between the cache 162, memory 158, persistent storage 170, communication unit 152, and input / output (I / O) interface 156. The communication fabric 140 can be implemented in any architecture designed to pass data or control information, or both, between the processor (such as a microprocessor, communication and network processor), system memory, peripherals, and other hardware components in the system. For example, the communication fabric 140 can be implemented on one or more buses.

[0030] Memory 158 and persistent storage 170 are computer-readable storage media. In this embodiment, memory 158 includes random access memory (RAM) 160. Generally, memory 158 can include any suitable volatile or non-volatile computer-readable storage media. Cache 162 is a high-speed memory that improves the performance of processor 154 by holding recently accessed data and data in the vicinity of recently accessed data from memory 158.

[0031] Program instructions and data used to implement embodiments of the present invention, such as program 175, are stored in persistent storage 170 via cache 162 for execution or access, or both, by one or more of the processors 154 of the server computer 150. In this embodiment, persistent storage 170 includes a magnetic hard disk drive. Alternatively, or in addition to a magnetic hard disk drive, persistent storage 170 may include a solid-state hard drive, a semiconductor storage device, read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, or any other computer-readable storage medium capable of storing program instructions or digital information.

[0032] The media used by the persistent storage device 170 may be removable. For example, a removable hard drive can be used as the persistent storage device 170. Other examples include optical discs, magnetic discs, thumb drives, and smart cards that are inserted into the drive for transfer to another computer-readable storage medium that is also part of the persistent storage device 170.

[0033] In these examples, the communication unit 152 provides communication with other data processing systems or devices, including the resources of client computing devices 104 and 110. In these embodiments, the communication unit 152 includes one or more network interface cards. The communication unit 152 can provide communication through the use of either or both physical communication links and wireless communication links. Software distribution programs and other programs and data used in implementing the present invention may be downloaded via the communication unit 152 to the persistent storage device 170 of the server computer 150.

[0034] The I / O interface 156 enables data input and output with other devices that may be connected to the server computer 150. For example, the I / O interface 156 can provide connection to an external device 190 such as a keyboard, keypad, touchscreen, microphone, digital camera, or other suitable input device or combination thereof. The external device 190 may also include, for example, a portable computer-readable storage medium such as a thumb drive, portable optical or magnetic disk, or memory card. Software and data used to implement embodiments of the present invention, for example, a program 175 on the server computer 150, can be stored on such a portable computer-readable storage medium and loaded into the persistent storage device 170 via the I / O interface 156. The I / O interface 156 also connects to the display 180.

[0035] The display 180 provides a mechanism for displaying data to the user and may be, for example, a computer monitor. The display 180 can also function as a touchscreen, such as the display of a tablet computer.

[0036] Figure 2 provides a flowchart 200 illustrating exemplary operation related to the implementation of this disclosure. In one embodiment, program 175 is started in response to a user connecting a client device 104 or 110 to program 175 via network 114. For example, program 175 is started in response to a user registering (e.g., opting in) an image forming apparatus (e.g., client device 104) with program 175 via a WLAN (e.g., network 114). In another embodiment, program 175 is a background application that continuously monitors the client device 104. For example, program 175 is a client-side application that is started when the user's image forming apparatus (e.g., client device 104) is started and monitors image data of the image forming apparatus.

[0037] After the program starts, in block 202, the method of program 175 generates a set of weighted labels. In one embodiment, program 175 generates a set of weighted labels and stores the set of weighted labels in the memory 158 of the server computer 150.

[0038] In block 204, the method of program 175 maps a set of weighted labels to a set of images. In one embodiment, program 175 maps a set of weighted labels in memory 158 of server computer 150 to a set of images in memory 158. In this embodiment, the set of images in memory 158 may initially contain no images, and program 175 maps the weighted labels to the target values ​​of the set of images of database objects in memory 158 received from client device 104.

[0039] In block 206, the method of program 175 receives multiple images from user equipment. In one embodiment, program 175 receives multiple images of a location from the user's client device 104. In this embodiment, program 175 stores the multiple images in memory 158. Furthermore, program 175 updates the multiple images from the client device 104 when program 175 receives additional images. In another embodiment, program 175 retrieves multiple images of a location from the client device 104. In this example, program 175 sends a request to transmit multiple images to the server computer 150 via the network 114.

[0040] In block 208, the method of program 175 identifies an object in one or more of the images. In one embodiment, program 175 uses a weighted set of labels in memory 158 to identify an object corresponding to a label in the set of labels for the multiple images received from client device 104. In this embodiment, program 175 utilizes computer vision techniques (e.g., machine learning models) to detect the object in the image corresponding to the label.

[0041] In block 210, the method of program 175 determines a score associated with one or more images. In one embodiment, program 175 uses the weights associated with each label in a set of weighted labels to determine a score for one or more images from a plurality of images in memory 158 received from client device 104. In this embodiment, program 175 determines a score for one or more sub-images of one or more of the plurality of images.

[0042] In block 212, the method of program 175 selects a base image. In one embodiment, program 175 determines the base image using one or more images from a plurality of images in memory 158 received from client device 104. In this embodiment, program 175 identifies the image in memory 158 with the lowest determined score and selects that image as the base image. In another embodiment, program 175 receives a user image from client device 104. In this embodiment, the user indicates to program 175 that the image is the base image for initial use.

[0043] In block 214, the method of program 175 generates a set of replacement images for a base image. In one embodiment, program 175 generates a set of replacement images in memory 158 for a base image using the score of the base image and the scores of one or more images from a plurality of images in memory 158 received from client device 104. In this embodiment, program 175 compares the score of the base image with the score of one or more images from the plurality of images to determine whether an image from the plurality of images can be included in the set of replacement images. Furthermore, the set of replacement images may include a set of replacement sub-images based on the scores of segments of the plurality of images.

[0044] In block 216, the method of program 175 determines whether the set of replacement images contains a suitable replacement image. In one embodiment, program 175 queries the set of replacement images in memory 158 to identify a replacement image for a base image. In this embodiment, program 175 identifies a replacement image by comparing the score of the base image with the score of an image in the set of replacement images. Program 175 can also identify a replacement subimage by comparing the score of a segment of the base image with the score of a subimage in the set of replacement subimages. Program 175 can identify replacements (e.g., subimages, images) by utilizing the lowest score, at least in part, based on a mapped set of weighted labels, or by identifying the label of the base image as the key of an object returned in a database.

[0045] In block 218, the method of program 175 performs a defined action. In one embodiment, program 175 performs a defined task. In this embodiment, if program 175 determines that the base image is to be replaced by a replacement image from the set of replacement images in memory 158, program 175 removes the base image from memory 158. Furthermore, program 175 may remove a segment of the base image from memory 158 that program 175 replaces with a replacement sub-image. In another embodiment, if program 175 determines that the set of replacement images in memory 158 does not contain a replacement image suitable for the base image, program 175 masks an object corresponding to an unsuitable condition in the base image. Furthermore, program 175 may mask a segment of the base image so that an object corresponding to an unsuitable condition in the base image is not visible to the user.

[0046] In block 220, the method of program 175 replaces the base image with an appropriate replacement image. In one embodiment, program 175 replaces the base image with a replacement image from a set of replacement images in memory 158. In this embodiment, program 175 replaces the base image with a returned replacement image which is determined to have a smaller score than the base image. In another embodiment, program 175 replaces a segment of the base image with a replacement sub-image which corresponds to the segment and is determined to have a smaller score than the segment of the base image. In this embodiment, program 175 returns a base image which includes the replacement sub-image, rather than the segment of the base image.

[0047] In an exemplary embodiment, the user intends to capture image data with client device 104 for a database of street images. The user intends to generate high-quality images of storefronts and houses and intends to avoid the following objects in the images: pedestrians (0.5), bicycles (0.5), buses (0.7), and overflowing trash cans (0.9) (e.g., a set of weighted labels). Program 175 assigns weights provided by the user to the objects, indicating how unsuitable they are. In this exemplary embodiment, as images are captured by client device 104, program 175 maps the set of weighted labels to the captured image data (i.e., map C is initially empty). The user walks down Main Street three times on three different days and captures three images of geolocation storefronts with client device 104. In the first image, there is a pedestrian on the left side of the storefront. In the second image, there is an overflowing trash can on the right side of the storefront. In the third image, the entire image of the storefront is obscured by a bus. Program 175 performs computer vision techniques on three images, assigning one labeled segment to each image (for example, "PEDESTRIAN", "OVERFLOWING_GARBAGE_CAN", and "BUS", respectively). Of the three images, Program 175 selects the first image (i.e., the image with the pedestrian) as the base image (for example, p').

[0048] Now, program 175 iterates through multiple images (e.g., a set of images "p") captured by client device 104. Program 175 determines that the first image equal to p' has an image of a pedestrian in the labeled segment of the first image (score 0.5). Program 175 determines that the second image has no object in the segment corresponding to the labeled segment of the base image (score 0.0 is assigned). Program 175 determines that the third image consists only of a bus in the segment corresponding to the labeled segment of the base image and is assigned a score 0.7. Of the three replacement candidates (including the base image as the "replacement"), program 175 determines that the second image has the lowest score and replaces the labeled segment of the base image with the segment corresponding to the labeled segment of the base image (i.e., a sub-image). Furthermore, program 175 removes the base image segment and inappropriate conditions from the database.

[0049] In another embodiment, the user intends to capture image data with client device 104 for a database of street images. The user intends to generate high-quality images of storefronts and houses and to avoid objects in the images that correspond to the user's competitors' content. In this embodiment, program 175 masks competitor content with another image if it appears in the database images. Furthermore, program 175 generates a set of labels for inappropriate content, including protected content from company 1 and protected content from company 2. In response to executing a map function (e.g., map C), program 175 returns [Company One --> {Image1.jpg, Image2.jpg, Image3.jpg}, Company Two --> {Image1.jpg, Image2.jpg, Image3.jpg}]. The user passes through the same area three times within a week. Program 175 identifies the protected content from company 1 and company 2 in the images from two passes. In the third pass, program 175 determines that the protected content of company 2 has been replaced with content of company 3, which is not a competitor of the user, and that company 3's content is not included in the label set "L". In this exemplary embodiment, program 175 selects the image from the third pass, which contains only the protected content of company 1, as the base image (p'). Furthermore, program 175 detects inappropriate content within each image and determines that there is no suitable image to replace the segment of the base image containing the inappropriate content. Thus, program 175 can mask or replace the segment of the base image containing the inappropriate content with another image.

[0050] This disclosure includes a detailed description of cloud computing, but the implementations of the teachings described herein are not limited to cloud computing environments. Rather, embodiments of the present invention can be implemented in any other type of computer environment that is currently known or may be developed in the future.

[0051] Cloud computing is a service delivery model that enables convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal administrative effort or interaction with service providers. This cloud model may include at least five characteristics, at least three service models, and at least four implementation models.

[0052] The characteristics are as follows:

[0053] On-demand self-service: Cloud consumers can unilaterally prepare computing power, such as server time and network storage, automatically as needed, without requiring human interaction with service providers.

[0054] Broad network access: Capabilities are available over the network and accessible via standard mechanisms. This facilitates use by heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, PDAs).

[0055] Resource pooling: A provider's computing resources are pooled and delivered to multiple consumers using a multi-tenant model. Various physical and virtual resources are dynamically allocated and reallocated as needed. Generally, consumers have a sense of location independence because they do not manage or know the exact location of the resources provided. However, consumers may be able to identify the location at a higher level of abstraction (e.g., country, state, data center).

[0056] Rapid Elasticity: Computing power can be prepared quickly and flexibly, allowing it to scale out automatically and immediately, and to be quickly released and scale in immediately. To consumers, the computing power available for preparation often appears unlimited and can be purchased in any quantity at any time.

[0057] Measured Services: Cloud systems leverage metric capabilities at a certain level of abstraction, appropriate for the type of service (e.g., storage, processing, bandwidth, active user accounts), to automatically control and optimize resource usage. Resource usage can be monitored, controlled, and reported, providing transparency to both service providers and consumers.

[0058] The service model is as follows:

[0059] Software as a Service (SaaS): The functionality offered to consumers is the ability to use the provider's applications running on a cloud infrastructure. These applications can be accessed from various client devices via thin client interfaces such as web browsers (e.g., webmail). Consumers do not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, storage, or even individual application functions, except for configuring a limited number of user-specific applications.

[0060] Platform as a Service (PaaS): The functionality offered to consumers is the ability to deploy applications they have created or acquired to cloud infrastructure using programming languages ​​and tools supported by the provider. Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, and storage, but they can control the deployed applications and, in some cases, the configuration of their hosting environment.

[0061] Infrastructure as a Service (IaaS): The functionality provided to consumers is the provision of processors, storage, networking, and other basic computing resources that enable consumers to deploy and run any software, including operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but they can control the operating system, storage, and deployed applications, and in some cases, partially control certain network components (e.g., host firewalls).

[0062] The deployment model is as follows:

[0063] Private Cloud: This cloud infrastructure is operated exclusively for a specific organization. This cloud infrastructure can be managed by that organization or a third party and can reside on-premises or off-premises.

[0064] Community Cloud: This cloud infrastructure is shared by multiple organizations to support a specific community with common interests (e.g., mission, security requirements, policies, and compliance). This cloud infrastructure can be managed by the organization or a third party and can reside on-premises or off-premises.

[0065] Public Cloud: This cloud infrastructure is provided to a large number of people or large industry groups and is owned by organizations that sell cloud services.

[0066] Hybrid Cloud: This cloud infrastructure combines two or more cloud models (private, community, or public). While maintaining the unique entities of each model, they are bound together by standards or individual technologies to achieve data and application portability (e.g., cloud bursting for load balancing across clouds).

[0067] Cloud computing environments are service-oriented environments that emphasize statelessness, low coupling, modularity, and semantic interoperability. At the core of cloud computing is the infrastructure, which includes a network of interconnected nodes.

[0068] Figure 3 shows an exemplary cloud computing environment 50. The cloud computing environment 50 includes one or more cloud computing nodes 10. Local computer devices used by cloud consumers (e.g., PDAs or mobile phones 54A, desktop computers 54B, laptop computers 54C, or automotive computer systems 54N, or a combination thereof) can communicate with these nodes. The nodes 10 can communicate with each other. The nodes 10 can be grouped physically or virtually (not shown) in one or more networks, such as the private, community, public, or hybrid clouds or a combination thereof. This allows the cloud computing environment 50 to provide infrastructure, platforms, or software as a service, or a combination thereof, without requiring cloud consumers to maintain resources on their local computer devices. Note that the types of computer devices 54A-N shown in Figure 3 are merely examples, and it should be understood that the computing nodes 10 and the cloud computing environment 50 can communicate with any type of electronic device via any type of network or network addressable connection (e.g., using a web browser) or both.

[0069] Figure 4 shows the functional abstraction model layer provided by the cloud computing environment 50 shown in Figure 3. It should be understood that the components, layers, and functions shown in Figure 4 are illustrative only, and the embodiments of the present invention are not limited to these. As illustrated, the following layers and corresponding functions are provided.

[0070] The hardware and software layer 60 includes hardware components and software components. Examples of hardware components include a mainframe 61, a reduced instruction set computer (RISC) architecture-based server 62, server 63, blade server 64, storage 65, and a network and network components 66. In some embodiments, the software components include network application server software 67 and database software 68.

[0071] The virtualization layer 70 provides an abstraction layer. From this layer, for example, the following virtual entities can be provided: virtual servers 71, virtual storage 72, virtual networks 73 including virtual private networks, virtual applications and operating systems 74, and virtual clients 75.

[0072] As an example, the management layer 80 can provide the following functions: Resource preparation 81 enables the dynamic procurement of computing resources and other resources used to perform tasks within the cloud computing environment. Metering and pricing 82 enables cost tracking as resources are used within the cloud computing environment and billing or invoicing for the consumption of these resources. As an example, these resources may include licenses for application software. Security enables not only protection of data and other resources but also identification and verification of cloud consumers and tasks. The user portal 83 provides consumers and system administrators with access to the cloud computing environment. Service level management 84 enables the allocation and management of cloud computing resources to ensure that requested service levels are met. Service Level Agreement (SLA) planning and execution 85 enables the pre-arrangement and procurement of cloud computing resources that are expected to be needed in the future in accordance with the SLA.

[0073] Workload layer 90 provides examples of the capabilities available in a cloud computing environment. Examples of workloads and capabilities available from this layer include mapping and navigation 91, software development and lifecycle management 92, virtual classroom education delivery 93, data analytics processing 94, transaction processing 95, and programs 175.

[0074] The present invention may be a system, method, or computer program product or combination thereof, integrated at any possible level of technical detail. The present invention can be beneficially implemented in any single or parallel system that processes instruction streams. The computer program product may include a computer-readable storage medium storing computer-readable program instructions for causing a processor to perform aspects of the present invention.

[0075] A computer-readable storage medium can be a tangible device capable of holding and storing instructions used by an instruction execution device. Examples of computer-readable storage media may be electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or appropriate combinations thereof. More specific examples of computer-readable storage media include portable computer diskettes, hard disks, RAM, ROM, EPROM (or flash memory), SRAM, CD-ROM, DVD, memory stick, floppy disk, punch cards, or grooved raised structures, and mechanically encoded devices on which instructions are recorded, and appropriate combinations thereof. Computer-readable storage devices as used herein should not be interpreted as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through optical fiber cables), or electrical signals transmitted through wires.

[0076] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or to an external computer or external storage device via a network (e.g., the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof). The network consists of copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. The network adapter card or network interface of each computing / processing device receives computer-readable program instructions from the network and transfers the computer-readable program instructions for storage on the computer-readable storage medium within each computing / processing device.

[0077] The computer-readable program instructions for performing the operation of the present invention may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for integrated circuits, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk and C++ and procedural programming languages ​​such as the C programming language or similar programming languages. The computer-readable program instructions are executable as a standalone software package, either entirely on the user's computer or partially on the user's computer. Alternatively, they may be executable partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or wide area network (WAN), or to an external computer (for example, via the Internet using an Internet service provider). In some embodiments, for example, an electronic circuit including a programmable logic circuit, a field-programmable gate array (FPGA), or a programmable logic array (PLA) can execute computer-readable program instructions by personalizing them using state information of computer-readable program instructions in order to perform aspects of the present invention.

[0078] Aspects of the present invention are described herein with reference to flowcharts or block diagrams, or both, of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It will be understood that each block in a flowchart or block diagram, or both, and any combination of blocks in a flowchart or block diagram, or both, can be implemented by computer-readable program instructions.

[0079] These computer-readable program instructions can be provided to a computer processor or other programmable data processing device to generate a machine, such that instructions executed via the processor of the computer or other programmable data processing device generate means for implementing functions / operations specified in one or more blocks of a flowchart or block diagram or both. These computer-readable program instructions can also be stored in a computer-readable storage medium that can be connected to a computer, a programmable data processing device, or other device or combination of devices that function in a particular way, such that the computer-readable program instructions on which the instructions are stored constitute one of the outputs containing instructions that implement a mode of function / operation specified in one or more blocks of a flowchart or block diagram or both.

[0080] Computer-readable program instructions, like instructions that perform a function / action specified in one or more blocks of a flowchart or block diagram or both on a computer, other programmable device, or other device, can also be loaded into a computer, other programmable data processing device, or other device and perform a series of operational steps on the computer, other programmable device, or other device to produce a computer-implemented process.

[0081] The flowcharts and block diagrams in the figures illustrate the configuration, function, and operation of executable implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or part of an instruction, which constitutes one or more executable instructions for implementing a specified logical function. In some alternative embodiments, the functions shown in the blocks may differ from the order shown in the figures. For example, two blocks shown consecutively may actually be achieved as a single step, executed simultaneously, substantially simultaneously, partially or entirely in overlapping time, or the blocks may be executed in reverse order depending on the functions involved. It should also be noted that each block in a block diagram or flowchart diagram, or both, and any combination of blocks in a block diagram or flowchart diagram, or both, can be implemented by a special-purpose hardware-based system that performs a specified function or operation, or a combination of special-purpose hardware and computer instructions.

[0082] In this specification, “one embodiment,” “an embodiment,” “an example embodiment,” or similar phrases indicate that certain features, structures, or characteristics described in relation to an embodiment may be included in the described embodiment, but not all embodiments necessarily include those features, structures, or characteristics. Furthermore, such expressions do not necessarily refer to the same embodiment. Moreover, where certain features, structures, or characteristics are described in relation to an embodiment, it is within the knowledge of those skilled in the art that such features, structures, or characteristics may be affected in relation to other embodiments, whether explicitly stated or not.

[0083] The terms used herein are for the purpose of describing specific embodiments and are not intended to limit the invention. In this specification, the singular forms “a,” “an,” and “the” include the plural form unless the context makes it clear otherwise. Furthermore, in this specification, the terms “comprises” or “comprising” or both identify the presence of a described feature, integer, process, operation, element or component or combination thereof, but do not preclude the presence or addition of one or more other features, integers, processes, operations, elements, components or groups thereof or combination thereof.

[0084] The descriptions of various embodiments of the present invention are presented for illustrative purposes only and are not intended to be exhaustive, nor are they intended to limit the disclosed embodiments. It will be apparent to those skilled in the art that many modifications and changes are possible without departing from the scope of the invention. The terms used herein have been selected to best describe the principles of the embodiments, their practical application to market-based technologies, or technical improvements, or to enable those skilled in the art to understand the embodiments described herein.

Claims

1. To generate a set of weighted labels, wherein one or more weighted labels in the set of weighted labels correspond to an inappropriate condition which is a condition which the user does not want to store in the database. Receiving multiple images of a location from the user's device, Identifying an object in one or more of the aforementioned plurality of images, wherein the object corresponds to the aforementioned inappropriate condition. Determining the score of one or more of the multiple images based at least partially on the identified object, Determining a base image from one or more of the aforementioned multiple images, wherein the base image is determined to be the image with the lowest determined score. Mapping the weighted set of labels to one or more images among the plurality of images, wherein the weighted labels correspond to the objects of the segments of the one or more images; A computer implementation method performed by a computer, comprising generating a set of replacement subimages for the location, based at least partially on the weighted labels of each segment of each of the one or more images among the plurality of images, wherein the replacement subimages correspond to each segment of the one or more images.

2. The method according to claim 1, further comprising replacing the segment of the base image with the replacement subimage of the set of replacement subimages in response to identifying a replacement subimage of the set of replacement subimages associated with a first determined score that is smaller than a second determined score associated with the base image.

3. The method of claim 2, further comprising removing the base image containing the object corresponding to the inappropriate condition from the database in response to replacing the segment of the base image with the replacement subimage of the set of replacement subimages.

4. The method according to claim 1, further comprising masking the segment of the base image in response to determining that there are no replacement subimages among the set associated with a first weighted label that are less than the value of a second weighted label associated with a segment of the base image.

5. The method according to claim 1, further comprising replacing the segment of the base image with the replacement subimage of the set of replacement subimages in response to identifying a replacement subimage of the set of replacement subimages that relates to a first weighted label value that is smaller than the value of a second weighted label related to the segment of the base image, wherein the segment of the base image includes the inappropriate condition.

6. The method of claim 5, further comprising removing the segment of the base image containing the unsuitable condition from the database in response to replacing the segment of the base image with the replacement subimage of the set of replacement subimages.

7. A computer, To generate a set of weighted labels such that one or more weighted labels in the set of weighted labels correspond to an inappropriate condition which is a condition that the user does not want stored in the database. Receiving multiple images of a location from the user's device, Identifying an object in one or more of the aforementioned plurality of images, wherein the object corresponds to the aforementioned inappropriate condition. Determining the score of one or more of the multiple images based at least partially on the identified object, Determining a base image from one or more of the aforementioned multiple images, wherein the base image is determined to be the image with the lowest determined score. Mapping the weighted set of labels to one or more images among the plurality of images, wherein the weighted labels correspond to the objects of the segments of the one or more images; A computer program that generates a set of replacement subimages for the location, based at least partially on the weighted labels of each segment of each of the one or more images among the plurality of images, wherein the replacement subimages correspond to each segment of the one or more images.

8. The computer program according to claim 7, further causing the computer to replace the segment of the base image with the replacement subimage of the set of replacement subimages in response to identifying a replacement subimage of the set of replacement subimages associated with a first determined score smaller than a second determined score associated with the base image.

9. The computer program according to claim 8, further causing the computer to remove from the database the base image containing the object corresponding to the inappropriate condition in response to replacing the segment of the base image with the replacement subimage of the set of replacement subimages.

10. The computer program according to claim 7, which in response to determining that there are no replacement subimages in the set associated with a first weighted label that are less than the value of a second weighted label associated with a segment of the base image, further causes the computer to mask the segment of the base image.

11. The computer program according to claim 7, further causing the computer to replace the segment of the base image with the replacement subimage of the set of replacement subimages in response to identifying a replacement subimage of the set of replacement subimages that is associated with a second weighted label value that is smaller than the value of a second weighted label associated with the segment of the base image, wherein the segment of the base image includes the inappropriate condition.

12. The computer program according to claim 11, further causing the computer to remove the segment of the base image containing the unsuitable condition from the database in response to replacing the segment of the base image with the replacement subimage of the set of replacement subimages.

13. One or more computer processors, One or more computer-readable storage devices, For execution by the one or more computer processors, the stored program instructions include: A program instruction to generate a set of weighted labels, wherein one or more weighted labels in the set of weighted labels correspond to an inappropriate condition, which is a condition that the user does not want stored in the database. A program instruction to receive multiple images of a location from a user device, A program instruction that identifies an object in one or more of the aforementioned plurality of images, wherein the object corresponds to a program instruction that matches the aforementioned inappropriate condition. A program instruction for determining the score of one or more of the multiple images based at least partially on the identified object, A program instruction for determining a base image from one or more of the aforementioned multiple images, wherein the base image is the image having the lowest determined score; A program instruction for mapping the weighted set of labels to one or more images among the plurality of images, wherein the weighted labels correspond to the objects of the segments of the one or more images, A computer system comprising: a program instruction for generating a set of replacement subimages for a location based at least partially on the weighted labels of each segment of each of the one or more images among the plurality of images, wherein the replacement subimage corresponds to each segment of the one or more images; and a program instruction for generating a set of replacement subimages for the location.

14. The computer system according to claim 13, wherein the stored program instructions further include a program instruction for replacing the segment of the base image with the replacement subimage of the set of replacement subimages in response to identifying a replacement subimage of the set of replacement subimages associated with a first determined score smaller than a second determined score associated with the base image.

15. The computer system according to claim 13, wherein the stored program instructions further include a program instruction for masking the segment of the base image in response to determining that there are no replacement subimages in the set associated with a first weighted label that are less than the value of a second weighted label associated with a segment of the base image.

16. The computer system according to claim 13, further comprising a program instruction which replaces the segment of the base image with the replacement subimage of the set of replacement subimages in response to identifying a replacement subimage of the set of replacement subimages that is associated with a value of a first weighted label that is smaller than the value of a second weighted label associated with a segment of the base image, wherein the segment of the base image includes the inappropriate condition.

17. The computer system according to claim 16, wherein the stored program instructions further include a program instruction for removing the segment of the base image containing the unsuitable condition from the database in response to replacing the segment of the base image with the replacement subimage of the set of replacement subimages.

Images