METHOD FOR SEARCHING FOR IMAGES USING ROTATIONAL GESTURE INPUT

Abstract

A computer-implemented method for searching for images, comprising: identifying a reference object in a two-dimensional reference image; determining a three-dimensional reference orientation axis of the reference object based on at least one attribute of the reference object, wherein the three-dimensional reference orientation axis extends in a direction from the front of the object through the center of mass of the object to the rear of the object; receiving an input from an input unit, wherein the input requests a change in a three-dimensional perspective of the reference object; determining a new orientation axis based on the input and the reference orientation axis; performing a search query on a set of two-dimensional images, wherein the search query is based on the new orientation axis and at least one attribute of the reference object;and displaying image search results that are ranked based on correlations with the new alignment axis and at least one attribute of the reference object.

Description

TECHNICAL AREA

[0001] The present disclosure relates to a search for images, more precisely a search for images using input of rotation gestures. BACKGROUND

[0002] Display units can show objects in two-dimensional (“2D”) or three-dimensional (“3D”) form. When an object is displayed in a 2D representation (e.g., in a flat photograph of a car), the data processing system lacks the information needed to change the view of the object to a different angle. On the other hand, objects displayed in a 3D representation can be viewed on a two-dimensional display unit from a variety of different angular orientations or perspectives. For example, CAD (computer-aided design) programs allow the user to rotate a filled (or wireframe) representation of an object to view it from different angles. The 3D model of the object includes additional information (e.g., dimensions, dimensions, and other details).Information about the X, Y, and Z axes allows the program to create a new image of the object when the user requests a change in perspective. However, for objects viewed in a 2D representation, a rotation gesture (or other form of user input) does not result in a different viewing angle. Therefore, the user cannot provide input to change the rotational viewing angle of the 2D images. Consequently, there is a need in engineering to solve the aforementioned problem. SUMMARY

[0003] From a first perspective, the present invention provides a method for searching for images, the method comprising: identifying a reference object in a two-dimensional reference image; determining a three-dimensional reference orientation axis of the reference object based on at least one attribute of the reference object; receiving an input requesting a change in a three-dimensional perspective of the reference object; determining a new orientation axis based on the input and the reference orientation axis; executing a search query in a set of two-dimensional images, the search query being based on the new orientation axis and the at least one attribute of the reference object; and displaying image search results that are ranked based on their correlation with the new orientation axis and the at least one attribute of the reference object.

[0004] From another perspective, the present invention provides a computer system comprising: a computer-readable storage medium on which program instructions are stored; and one or more processors configured to execute the program instructions to perform a method comprising: identifying a reference object in a two-dimensional reference image; determining a three-dimensional reference orientation axis of the reference object based on at least one attribute of the reference object; receiving an input requesting a change in a three-dimensional perspective of the reference object; and determining a new orientation axis based on the input and the reference orientation axis.Executing a search query in a set of two-dimensional images, wherein the search query is based on the new orientation axis and the at least one attribute of the reference object; and displaying image search results that are ranked based on the correlation with the new orientation axis and the at least one attribute of the reference object.

[0005] From another perspective, the present invention provides a computer program product for searching for images, wherein the computer program product comprises: a computer-readable storage medium that is readable by a processing circuit and stores instructions for execution by the processing circuit to carry out a method for performing the steps of the invention.

[0006] From another perspective, the present invention provides a computer program that is stored on a computer-readable medium and can be loaded into the internal memory of a digital computer, and which includes software code sections that, when the program is executed on a computer, perform the steps of the invention.

[0007] From another perspective, the present invention provides a computer program product for implementing a method for searching for images, wherein the computer program product comprises a computer-readable storage medium in which program instructions are embodied, wherein the program instructions are executable by at least one computer processor to cause the computer processor to: identify a reference object in a two-dimensional reference image; determine a three-dimensional reference orientation axis of the reference object based on at least one attribute of the reference object; receive an input requesting a change in a three-dimensional perspective of the reference object; determine a new orientation axis based on the input and the reference orientation axis;Executing a search query in a set of two-dimensional images, wherein the search query is based on the new orientation axis and the at least one attribute of the reference object; and displaying image search results that are ranked based on the correlation with the new orientation axis and the at least one attribute of the reference object.

[0008] Embodiments of the present disclosure relate to a method for searching for images. The method comprises identifying an object in a two-dimensional reference image. The method comprises determining a three-dimensional reference orientation axis of the object based on at least one attribute of the object. The method comprises receiving an input requesting a change to a three-dimensional perspective of the object. The method further comprises determining a new orientation axis based on the input and the reference orientation axis. The method comprises executing a search query in a set of two-dimensional images, wherein the search query is based on the new orientation axis and the at least one attribute of the reference object.The procedure also includes displaying image search results that are ranked based on correlations with the new alignment axis and the object's attribute.

[0009] Other embodiments of the present disclosure relate to a computer system and a computer program product for carrying out the method.

[0010] The above summary is not intended to describe every illustrated embodiment or every implementation of the present disclosure. List of characters

[0011] The drawings included in this application are incorporated into the specification and form part of it. They show embodiments of the present disclosure and, together with the description, explain the basic concepts of the disclosure. The drawings are intended solely to illustrate certain embodiments and do not constitute a limitation of the disclosure. Fig. Figure 1 shows a block diagram of a processing system according to the embodiments. Fig. Figure 2 shows a block diagram of an exemplary cloud computing environment with one or more data processing nodes with which local data processing units used by cloud customers exchange data according to embodiments. Fig. Figure 3 shows a block diagram of a set of functional abstraction layers provided by a cloud computing environment according to embodiments. Fig. Figure 4 shows a flowchart for a method for image search according to embodiments. Fig. 5A and Fig. Figure 5B shows an example of an object that, according to one embodiment, is displayed in two different visual orientations. Fig. Figure 6 shows an example of an object and a user's rotation gesture, according to one embodiment. Fig. Figure 7 shows an example of an object and a spherical model which, according to one embodiment, displays the available viewing angles in the image search results. Fig. Figure 8 shows an example of a spherical model which, according to one embodiment, displays the available viewing angles in the image search results and indicates the degree of correlation between the available images and the reference image. DETAILED DESCRIPTION

[0012] Image search engines do not provide 2D image results based on a specific viewpoint of an object within the image, nor do they allow users to specify a particular viewing angle. However, it is desirable to be able to view certain objects from different perspectives as 2D images when such objects cannot be represented by a 3D image model.

[0013] The embodiments described herein provide systems, methods, and computer program products that allow users to input a desired change in the viewing angle of an object in a 2D image and obtain 2D image search results that represent the desired viewing angle. Furthermore, the embodiments described herein enable the crowdsourcing of new images for missing viewing angles based on user demand to view specific objects from particular perspectives.

[0014] Referring to the drawings in which identical or similar elements are represented by identical numbers, and initially to Fig. Figure 1 shows an exemplary processing system 100 according to an embodiment on which the present embodiments can be carried out. The processing system 100 comprises at least one processor (CPU) 104, which is functionally connected to other components via a system bus 102. A cache 106, a read-only memory (ROM) 108, a random access memory (RAM) 110, an input / output (I / O) adapter 120, a sound adapter 130, a network adapter 140, a user interface adapter 150, and a display adapter 160 are functionally connected to the system bus 102.

[0015] A first storage unit 122 and a second storage unit 124 are functionally connected to the system bus 102 via the I / O adapter 120. Storage units 122 and 124 can be disk storage units (e.g., magnetic or optical disk storage units), magnetic semiconductor units, and so on. Storage units 122 and 124 can be the same type of storage unit or different types of storage units.

[0016] A loudspeaker 132 is functionally connected to the system bus 102 via the sound adapter 130. A transceiver 142 is functionally connected to the system bus 102 via the network adapter 140. A display unit 162 is functionally connected to the system bus 102 via the display adapter 160.

[0017] A first user input unit 152, a second user input unit 154, and a third user input unit 156 are functionally connected to the system bus 102 via the user interface adapter 150. The user input units 152, 154, and 156 can be a keyboard, a mouse, a keypad, an image capture unit, a motion detection unit, a microphone, a unit incorporating the functionality of at least two of the aforementioned units, or other suitable types of input units. The user input units 152, 154, and 156 can be the same type of user input unit or different types of user input units. The user input units 152, 154, and 156 are used for inputting and outputting information to and from the system 100.

[0018] An image analysis component 172 is functionally connected to the system bus 102. The image analysis component 172 (or routine) identifies objects in an image based on image analysis, image processing, metrology, edge detection, object recognition, classification, etc., performed on the image. The image analysis component 172 is configured to identify a variety of different objects based on numerous object attributes. These attributes can include color, model, type, shape, size, and so on. Furthermore, the image objects are classified using appropriate image classification procedures based on the identified attributes.

[0019] An image search engine component 174 is functionally connected to the system bus 102. The image search engine component 174 searches for images based on keywords, an image itself, a web link to an image, image metadata, the distribution of colors, shapes, the rotation angle, etc.

[0020] The processing system 100 may also include other (not shown) elements, as a person skilled in the art can readily consider, and certain elements may also be omitted. For example, the processing system 100 may include various other input and / or output units, depending on its specific implementation, as a person skilled in the art will readily understand. For example, different types of wireless and / or wired input and / or output units may be used. Furthermore, additional processors, controllers, memory, etc., may also be used in various configurations, as a person skilled in the art will readily recognize. These and other variations of the processing system 100 are readily apparent to a person skilled in the art in light of the teachings provided in this disclosure.

[0021] It should be made clear from the outset that the implementation of the teachings set forth herein is not limited to a cloud computing environment, although this disclosure includes a detailed description of cloud computing. Instead, embodiments of the present disclosure can be implemented together with any type of data processing environment, now known or hereafter invented.

[0022] Cloud computing is a service delivery model that enables seamless, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, main memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with a service provider.

[0023] A cloud computing environment is service-oriented, focusing on statelessness, low coupling, modularity, and semantic interoperability. At the heart of cloud computing lies an infrastructure that comprises a network of interconnected nodes.

[0024] With reference to Fig. Figure 2 illustrates the cloud computing environment 250. As shown, the cloud computing environment 250 has one or more cloud computing nodes 210 with which local data processing units used by cloud users, such as the electronic assistant (PDA, personal digital assistant) or mobile phone 254A, the desktop computer 254B, the laptop computer 254C, and / or the automotive computer system 254N, can exchange data. The nodes 210 can exchange data with each other. They can be grouped physically or virtually into one or more networks, such as private, community, public, or hybrid clouds (not shown), as described above, or into a combination thereof. This enables the cloud computing environment 250 to offer infrastructure, platforms, and / or software as a service, for which a cloud user does not need to maintain resources on a local data processing unit.It should be noted that the types of in . Fig. The data processing units 254A to N shown are for illustrative purposes only, and the data processing nodes 210 and the cloud computing environment 250 can exchange data with any type of computer unit via any type of network and / or any type of network-accessible connection (e.g., using a web browser).

[0025] With reference to Fig. 3, a set of functional abstraction layers is shown, which are used by the cloud computing environment 250 ( Fig. 2) be provided. It should be clear from the outset that the in Fig. The components, layers, and functions shown in the three diagrams are for illustrative purposes only, and embodiments of the invention are not limited to them. As shown, the following layers and corresponding functions are provided:

[0026] A hardware and software layer 360 comprises hardware and software components. Examples of hardware components include: mainframe computers 361; servers based on the RISC (Reduced Instruction Set Computer) architecture 362; servers 363; blade servers 364; storage units 365; and networks and network components 366. In some embodiments, software components include network application server software 367 and database software 368.

[0027] The virtualization layer 370 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual servers 371; virtual storage 372; virtual networks 373, including virtual private networks, virtual applications and operating systems 374; and virtual clients 375.

[0028] In one example, the management layer 380 can provide the functions described below. Resource provisioning 381 provides the dynamic procurement of data processing resources and other resources used to perform tasks within the cloud computing environment. Metering and pricing 382 provides cost tracking for the use of resources within the cloud computing environment, as well as billing or invoicing for the use of these resources. In one example, these resources might include application software licenses. Security provides identity verification for cloud users and tasks, as well as protection for data and other resources. A user portal 383 provides users and system administrators with access to the cloud computing environment.Service scope management (384) provides the allocation and management of cloud computing resources so that the required service goals are met. Service level agreement (SLA) planning and fulfillment (385) provides the advance planning and procurement of cloud computing resources for which a future requirement is anticipated, in accordance with an SLA.

[0029] A workload layer 390 provides examples of the functionality for which the cloud computing environment can be used. Examples of workloads and functions that can be provided by this layer include: mapping and navigation 391; software development and lifecycle management 392; delivery of training in virtual classrooms 393; data analytics processing 394; transaction processing 395; and image search engine processing 396.

[0030] In certain embodiments, an image search engine receives input from a user regarding a change in the viewing angle of a 2D object in an image and searches for an existing image stored on a memory unit that corresponds to this new perspective. Thus, there is a collection of different 2D images of the object stored from multiple different viewing angles. This differs from the mechanism for changing the viewing angle of a three-dimensional object (e.g., an object created in a CAD program), where the system programmatically generates a completely new image each time the viewing angle and / or zoom level changes.

[0031] With reference to Fig. In certain embodiments, an image search engine identifies an object in an image in step 400. In step 402, the search engine determines an orientation vector of the identified object. In step 404, the search engine receives user input to rotate the object in the image. In an example, which is explained in more detail here, the user input may be a rotation gesture made by the user's hand. Based on the user input and the magnitude of the change in the orientation vector, the image search engine searches a set of images to determine if other 2D images of the object with the new orientation vector are available. In step 406, the search engine performs a search query based on the new orientation vector and on one or more attributes of the image (e.g., the object's color).If the search engine determines in step 408 that another 2D image is available, this new image is displayed to the user on the display unit in step 410. If the search engine determines in step 408 that no 2D images are available for the new alignment vector, there are no images to display to the user. The search engine recognizes that information is missing, and in step 412, it makes a request for the missing information from one or more sources. As explained in more detail here, in certain embodiments, the request for additional information may be a crowdsourcing request. Crowdsourcing involves asking a large number of people to provide their own images for the image search engine.These provided images help to complete a set of images, allowing users to view the object from any 360-degree rotational perspective. After step 410 or step 412 is executed, the system returns to step 404 so the user can input another rotation gesture. In certain embodiments, the image database can also be updated with the new image (e.g., automatically or after image release by a system administrator) if the request for a missing image in step 412 is met by an external source providing such an image, after which future searches are performed based on the updated image database.

[0032] In certain implementations, the user is shown a 2D image containing one or more objects. In step 400, the image search engine analyzes the 2D image to identify the objects present. Object identification is based on image analysis, image processing, metrology, edge detection, object recognition, classification, and other processes performed on the image. An image analysis routine can identify a variety of different objects based on numerous object attributes. These attributes can include color, model, type, shape, size, and so on. Furthermore, the image objects are classified using appropriate image classification methods based on the identified attributes.

[0033] After the search engine identifies the presence of the object, in step 402 it determines the orientation of the objects. Specifically, the search engine identifies a basic orientation (or orientation vector) for each of the objects in the image. In one embodiment, the orientation vector is an axis passing through the calculated center of mass of the object, and this vector extends in a direction from the front of the object to the back of the object. This initial orientation vector of the object serves as a reference that, together with the user's rotation gesture, is used to compute a new orientation vector. In one embodiment, the initial orientation vectors of the objects present in the 2D image are part of the image's metadata and are used by the search engine when searching for new images.It should be noted that in certain embodiments, the orientation vector does not pass through the center of mass of the object and extend from a front to a back, but may run in a different direction relative to the object.

[0034] In certain embodiments, a library containing various orientation directions is available for different object classification types. In these embodiments, the data in the library indicates a reference orientation for an animal (e.g., the front of its head), a car (e.g., the front of the car), a house (e.g., the side of the building with the entrance door), or another classification type. Once a reference side of the image object has been determined in these embodiments, an image analysis module assigns the reference orientation axis to the image object. The assigned axis is the reference axis for the recognized object. Once the reference axis of an image object is identified, other image objects present in the same or a different image are compared to it, and a relative reference orientation or side is calculated.The relative orientation of various image objects is compared to the reference image object, and the orientation axis is calculated based on the difference. In one embodiment, the orientation of various image objects relative to the reference image object is calculated and assigned as the current angular orientation of the object. The angular orientation of the object is calculated relative to a 3D axis, and accordingly, the angular orientation of the 3D axis is calculated and belongs to the metadata of the image object.

[0035] With reference to Fig. 5A and Fig. 5B In certain embodiments, a 2D image (e.g., a digital image of a photograph) containing at least one object (e.g., a car) is displayed on a display unit. The image is a 2D image, and the object is displayed to the user from an initial three-dimensional viewpoint (i.e., perspectively). With reference to Fig. 5A provides a reference image viewing angle for an object 500. In which in Fig. In the example shown in Figure 5A, the reference image is a front view of object 500, which can be aligned on the Z and X axes. However, it should be noted that any other suitable viewing angle can be used as the reference viewing angle instead of a front view. The reference viewing angle also has a corresponding reference orientation vector that is perpendicular to the other two axes. With reference to Fig. Figure 5A shows an X-axis 504 and a Z-axis 502, extending in horizontal and vertical directions, and the reference alignment is along the Y-axis 506, which runs perpendicular to the other axes (see also the Y-axis 506 in Fig. 5B). In one embodiment, the Y-axis 506 is the orientation axis (or orientation vector 506) of the object, and the orientation vector 506 passes through a center of mass of the object from a front of the object to a back. With reference to Fig. In 5B, object 500 has an orientation vector 506 that points forward, to the left, and slightly downward at an angle. The direction of the orientation vector 506 in Fig. 5B exhibits a specific angular offset relative to the reference alignment vector in each of the directions of the X-axis 504, the Y-axis 506 and the Z-axis 502. Fig. 5A on.

[0036] With reference to Fig. 6 In one embodiment, the user can request a different viewing angle if they wish to see the object 600 in a different orientation or perspective by providing input to the image search engine. In response to a rotation gesture or other input from the user (e.g., a rotation gesture 604 of the user's hand 602), an image search engine determines the extent of the desired change in the rotational viewing angle of the object 600 in a 2D image and searches for another 2D image that corresponds to the new viewing angle requested by the user. For example, as in Fig. 6 is shown, a rotation gesture 604 is performed, and an angle change 606 is requested for the perspective of the object 600. Again with reference to Fig. 5B will present object 500 to the user from a different perspective relative to the one in Fig. 5A or Fig. The reference viewing angle shown is displayed in 6. The user can request changes to the viewing angle by providing input via gestures as often as desired. In certain implementations, the search engine triggers a request to one or more resources to provide the missing image information if no other 2D image is available for the new rotational viewing angle.

[0037] The input of rotation gestures that the user provides to the search engine to change the rotational viewing angle may vary depending on the type of input device(s) available to the user.

[0038] In one embodiment where the input unit is a mouse, the user can perform a click-and-drag operation to request a new perspective of the object. For example, the click-and-drag operation can specify a distance and angle relative to an origin point (e.g., the position where the mouse was clicked, or the mousedown event). The distance and angle of the final mouse position (e.g., the position where the mouse was released, or the mouseup event) from the origin point is a vector that can be used to specify a desired rotation of the object. In another example, the object is in a central position, and a distance between this central position and the positions of the mousedown and mouseup events is used to specify a desired rotation of the object.

[0039] In an embodiment where the input unit is a touchscreen display unit, the user can perform an operation by touching, dragging, and rotating with one or more fingers to request a change in orientation. With one finger, this rotation request can be performed in the same way as the click-and-drag input described above with a mouse. With more than one finger, the user can pinch or spread the touchscreen to perform a further operation for zooming in or out.

[0040] In one embodiment where the input unit is an augmented reality (or virtual reality) display unit, the user can perform a rotation gesture (e.g., by rotating a hand) to request a change in orientation. In certain embodiments, the user can perform a 3D rotation gesture relative to the 2D object based on data from the Internet of Things (IoT) or other camera images, and the system can use this information to identify the direction and angular movement of the gesture. For example, cameras can photograph a person's hand and, through image processing, determine changes in the hand's position, size, and orientation. The rotation gesture is then determined based on these changes.It should be noted that any other suitable type of input can be provided by the user, provided that the input conveys the desired change in the object's viewing angle.

[0041] In one embodiment, the search engine, in response to a rotation gesture (or other suitable input) from a user, uses: the direction of the rotation gesture; and / or the angular movement of the rotation gesture; and / or the extent of the rotation gesture; and / or the angular orientation (i.e., the orientation vector) of the object currently displayed in the 2D image. The input thus specifies a change in the orientation vector relative to a current orientation vector. Based on this input data, the search engine calculates a new orientation vector of the object, and then creates a new search query based on this data. The search engine executes the search query and returns one or more images of the object whose orientation vector matches (or at least closely matches) the calculated new orientation vector. This new image is displayed to the user on a display unit.In this way, the user can easily see how the object looks from a different perspective using a collection of various 2D images, without relying on complex 3D object manipulation programs (e.g., CAD programs). Furthermore, the object may not be suitable for representation as a three-dimensional object in a CAD program (e.g., a mountain range, a tourist landmark, houses, buildings, etc.). This process of selecting new perspectives can be repeated by the user as often as needed to view the object from different angles.

[0042] In certain implementations, the search engine uses one or more attributes of the object identified in the 2D image to perform a search query. As mentioned above, the search engine performs image analysis on the objects in the 2D image to determine a basic orientation vector of the object. However, the image analysis can also be used to identify one or more additional attributes of the object to aid the search query. Furthermore, the image file's metadata can be used to support the search query. Examples of other attributes include the object's size, color(s), type or category, geographic location where the image was taken, a timestamp indicating when the image was taken, a product part number, and other image file metadata such as image resolution, graphical characteristics of the object, and so on.

[0043] In one embodiment, in response to a user's rotation gesture, the search engine executes a search query to find a new 2D image of the object that has certain attributes that correlate with those of the original 2D image. For example, if the user is viewing an image of a red sedan from a first angle and then performs a rotation gesture to see a new angle, it would be less effective for the user if the search engine displayed a new 2D image of a blue van. While both are automobiles, they look quite different. The user might then feel that the connection to the original image has been broken if the new image does not at least substantially resemble the original.Based on these identified attributes, the search engine attempts to find a new 2D image with the new orientation that has similar (or the same) visual attributes as the original 2D image. Thus, while the viewer doesn't actually see different views of the same 3D object, it should feel more natural to the user to see different 2D images with very similar visual characteristics.

[0044] In certain implementations, the search engine requires that all 2D images depict the exact same object. For example, if a company has a website with an online product catalog, it might want to ensure that the different views of the object are not of slightly different products, to avoid confusing customers when making a purchase. For instance, a product part number (or serial number) could be an image metadata attribute of the various 2D images, and this would help ensure that the image search results do not return a view of a different product.

[0045] When the search engine displays a 2D image search result, in certain implementations it also shows the user which other views are available. There may be situations where the search engine finds many stored images of an object with front-facing views, but the product's image collection contains few (or no) rear views. The system visually indicates the availability of 2D images to the user, allowing them to make an informed decision about which rotational view to select next. For example, if the user knows that no rear views of the 2D object are available in the collection, they will not waste time performing a rotation gesture in the direction of the rear view.

[0046] With reference to Fig. 7 In one embodiment, the user is shown a visual representation of the available views of an object 704 in the image search results in the form of a three-dimensional sphere 706. In this example, the sphere 706 has shaded areas that indicate the presence of a 2D image of the object 704 from different viewing angles. As in Fig. As shown in Figure 7, the 3D wireframe representation of the sphere 706 has an orientation vector 704. Before the user initiates a rotation gesture, a display unit shows a current viewpoint of the object 702 with an orientation vector 704 that is identical to the orientation vector 704 of the sphere 706. In certain embodiments, the user can interact with an input unit by means of a rotation gesture, and the image search engine causes the sphere 706 to rotate on the display unit. The new orientation vector 704 of the sphere 706 provides the user with a preview of what the new viewpoint of the object 702 will look like. In certain embodiments, the image search engine updates the 2D image of the object 702 only after the user has performed the rotation gesture.In other embodiments, the image search engine continuously searches for images of the object and updates them (if available) while the user continues to rotate the sphere 706.

[0047] In the Fig. In the example shown, sections of the wireframe sphere 706 are shaded in binary form (e.g., black sections 708 and white sections 710) to indicate whether 2D images are available for these viewpoints. In this example, if the user were to provide a rotation gesture for the sphere 706, it would rotate to reveal a new front section of the sphere. If this most forward-facing section of the sphere is shaded black, this would mean that an image search for this viewpoint would yield no results. Therefore, the user would know to continue the rotation gesture to a different viewpoint that would produce an image result. In one embodiment, if the user rotates the sphere 706 to a viewpoint that does not yield any image results, the image search will find results for a different viewpoint that most closely matches the user's request.It should be noted that the foremost section of the sphere does not necessarily have to be the section corresponding to the new viewing angle; any suitable section can be used. In certain embodiments, the section of sphere 706 corresponding to the newly selected viewing angle is highlighted, colored, or otherwise indicated to the user.

[0048] In certain embodiments, the wireframe sphere sections are not only displayed to the user in binary form, but also feature additional visual cues or markings. For example, if the original 2D image of the object is a blue sedan, several different viewing angles may show different 2D images of a blue sedan. In this case, the wireframe sphere sections corresponding to these viewing angles may be colored blue. However, at other viewing angles, only images of a red sedan may be present. For these viewing angles, the wireframe sphere sections corresponding to these viewing angles may be colored red. This would alert the user that while they are still seeing a sedan at these angles, it is not the same color as the original 2D image of the object.It should be noted that the type of indication or marking on the sphere is not limited to different colors. The type of marking can represent another visual attribute of the object (e.g., size, type, height, etc.).

[0049] With reference to Fig. 8 In certain embodiments, the type of marking may also be a suitable visual representation indicating how closely the object corresponds to the original 2D image of the object in the various views. In one embodiment, as described in Fig. Figure 8 shows a heat map of the sphere 800, even if a 2D image of the object exists for all the different three-dimensional viewing angles, representing the degree of agreement (or deviation) of the objects in the different viewing angles with the original image of the object. With reference to Fig. Figure 8 represents the sphere 800 with the orientation vector 802. In this example, the sphere 800 has multiple surface sections. Some of these sections are white sections 804 (or transparent sections), indicating that no image exists for that viewpoint. Other sections are various shades of gray. A light gray section 806 indicates that while an image exists for that viewpoint, the visual features of the object in that image do not closely match the original image. The degree of darkness of the shading in a section indicates how closely an existing image matches the original image. In this respect, the black sections 808 match the original image much more closely than the lighter gray sections (e.g., the gray section 806).This heat map of the Sphere 800 provides the user with an easy-to-understand map on which he can see where there are closely matching images.

[0050] In certain embodiments, viewing angles that closely match the original image (e.g., having the same color and size) can be displayed in one color on the 360-degree heat map, while other viewing angles with a lower degree of match (e.g., if they meet the criteria of the requested viewing angle but differ from the original image in color and / or size) can be displayed in a different color on the 360-degree heat map (e.g., shown as red on the heat map as opposed to green). It should be noted that the Sphere 800 (or other display) does not necessarily have to be divided into individual surface sections, and that other suitable visual displays can be used to represent a heat map and the degree of match between the various existing 2D images and the original image.

[0051] In other embodiments, a new angular orientation is determined based on the user's gesture and the angular orientation of the original image object, and the image search results are ranked based on their deviation from this new angular orientation. When search results are displayed, the image search engine analyzes the attributes of the image object and identifies the availability of other images from various sources.

[0052] It should be noted that the viewpoint indicator does not necessarily have to be a 3D wireframe sphere. It can also be any other suitable object or visual display that allows the user to select a different three-dimensional viewpoint by means of a rotation gesture (or other input method). For example, if the object is a cow, the visual display could be a 3D wireframe representation of a cow or another object with a suitable shape. Furthermore, the visual display does not have to be a 3D wireframe model. It can be any other suitable three-dimensional shape with various surface markings indicating the number (or absence) of image search results.For example, the visual display could be a smooth sphere with a heat map (or color gradient) indicating the availability of search results from different viewing angles. In other embodiments, the visual display is a 2D representation rather than a three-dimensional model.

[0053] In certain embodiments, the search engine implements procedures to identify, request, and supplement missing image information (or to request better image information). There may be cases where a complete set of 2D images for every possible viewpoint of a particular object is not available. In response to a rotation gesture and an image search query that returns no results, the search engine triggers a request for additional image data from one or more recipients. In one embodiment, the request to one or more recipients includes measures for image procurement through crowdsourcing. Crowdsourcing generally refers to a procurement model in which individuals or organizations obtain goods and services, such as ideas and funding, from a large and dynamic group of contributors (e.g., internet users).Crowdsourcing distributes the effort among many participants to achieve a cumulative result. In the present embodiments, the goal is to obtain an extremely large number of relevant images for a wide variety of different objects and in a multitude of different orientations. Creating such a large collection of 2D images can therefore be very time-consuming, and crowdsourcing can reduce or mitigate this effort. If the target object is, for example, a tourist attraction like the Eiffel Tower, tourists might be willing to submit their vacation photographs of the tower from many different angles. In this way, a highly relevant set of 2D images of the tower from a variety of perspectives could be created. In certain embodiments, the crowdsourcing contributors are the general public (e.g.,(All internet users). In other embodiments, the crowdsourcing contributors are a more limited group of users, for example, the employees of a company. In one embodiment, the provider of the original content provides contributors who offer alternative perspectives of the main image with compensation (e.g., in the form of money, discounts on products / services, etc.) for their image contributions.

[0054] In one embodiment, the trigger for creating a crowdsourcing request for images occurs when a user performs a rotation gesture to view the object from a specific angle (provided that no image exists for that angle). As above regarding Fig.As explained in section 4, the image search engine sends a request for missing image information in step 410 if step 406 determines that no image exists for the new rotation vector. In other words, if a user has requested to see an object from a particular perspective, this indicates that there is indeed a demand to view the object from that angle. However, if there has never been a request to see a product from, say, the back, it might not be worthwhile to crowdsource photographs of the back of that object. In an example where the object is a car, there might be little or no need to see the underside of the car. In this example, users are not interested in viewing the object from that angle.

[0055] In one embodiment, the owner of the parent image (i.e., the reference image) has the opportunity to review the image when contributors respond to the request and offer additional images. The owner of the reference image can review the contributed images to determine whether they are of sufficient quality or adequately match the visual attributes of the parent image. If the owner of the parent image decides that the contributed images are suitable, they can save the image to a storage device and make additional viewpoints of the image available for viewing. In another embodiment, the parent image may not have an owner, and there is no person to manually review the contributed images. In this embodiment, image processing is performed on the contributed image to determine whether it adequately matches the parent image.

[0056] In one embodiment, even if the search query yields some results, the image search engine can trigger a request for additional image data if the image match quality for the relevant rotational view is low. For example, a 2D image of an object for a particular viewpoint might exist that has a very low correlation with the object in the original reference image. For instance, the original reference image might be an image of a gray, short-bed pickup truck, while the stored image for the other perspective might be an image of a brown, long-bed pickup truck where the make and model do not match the original. In this example, the brown, long-bed pickup truck might have some visual attributes that meet a certain threshold to be considered a sufficient match.However, this brown truck does not match the original image very well. In this embodiment, in response to a rotation gesture and a finding that a stored image has a low correlation with the reference image for the new orientation vector (i.e., is below a certain correlation threshold), the search engine sends a request to one or more contributors to update the image and improve the quality of the match. After receiving a new image from a contributor in response to the request, the system compares the newly received image with the previous image. If the new image is found to have a better match (i.e., correlates better with the attributes of the original reference image than the current image), the image search replaces the current image with the new image.If, in an example, the reference orientation axis in the received image and at least one attribute of the object in the received image match the reference orientation axis and at least one attribute of the reference image to a predetermined degree (e.g., this could be specified by the content provider), the system adds the received image to the set of two-dimensional images. By accepting new images in this way, which may replace older ones, the 360-degree rotation collection of 2D images for a given object can be continuously improved over time.

[0057] In one embodiment, the system uses a dynamic, context-aware, relational context redrawing based on usage type, task, object session goal, and logical history of the object with respect to: historical machine learning regarding local users; cloud-based usage for different unique users (shared context); user-crowded machine learning based on object orientation; and prediction of user behavior patterns when rotating objects.

[0058] In one embodiment, the system uses a physical location enhancement through crowdsourcing, geographically identifying individuals most likely to contribute the provided content to complete a model or goal. For example, if the area is defined by geofencing, the system selectively considers each person's current dynamic position and decides whether their unique perspective would be beneficial for that viewpoint and time (i.e., a temporal decision). This process selectively includes only specific individuals within certain marked locations and focuses on one or more temporal events or a timeline, resulting in a unique and more complete model.

[0059] The present invention may be a system, a method, and / or a computer program product at any possible level of technical integration. The computer program product may comprise a computer-readable storage medium (or media) on which computer-readable program instructions are stored to induce a processor to execute aspects of the present invention.

[0060] A computer-readable storage medium can be a physical unit capable of retaining and storing instructions for use by a system to execute instructions. For example, a computer-readable storage medium can be an electronic storage unit, a magnetic storage unit, an optical storage unit, an electromagnetic storage unit, a semiconductor storage unit, or any suitable combination thereof, without limitation. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), and erasable programmable read-only memory (EPROM).Flash memory), static random-access memory (SRAM), portable compact storage disk-read-only memory (CD-ROM), a DVD (digital versatile disc), a memory stick, a floppy disk, a mechanically coded unit such as punched cards or raised structures in a groove on which instructions are stored, and any suitable combination thereof. A computer-readable storage medium shall not, in its use herein, be understood as volatile signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses traveling through an optical fiber cable), or electrical signals transmitted by a wire.

[0061] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to individual data processing units or, via a network such as the internet, a local area network, a wide area network, and / or a wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission lines, wireless transmission, routing computers, firewalls, switching units, gateway computers, and / or edge servers. A network adapter card or network interface in each data processing unit receives computer-readable program instructions from the network and forwards them for storage on a computer-readable storage medium within the respective data processing unit.

[0062] Computer-readable program instructions for executing work steps of the present invention may be assembler instructions, ISA (Instruction Set Architecture) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, configuration data for integrated circuits, or either source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., as well as conventional procedural programming languages ​​such as the programming language "C" or similar programming languages.The computer-readable program instructions can be executed entirely on the user's computer, partially on the user's computer as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In the latter case, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, via the internet using an internet service provider).In some embodiments, electronic circuits, including, for example, programmable logic circuits, field programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), can execute computer-readable program instructions by using state information from the computer-readable program instructions to personalize the electronic circuits to implement aspects of the present invention.

[0063] Aspects of the present invention are described herein with reference to flowcharts and / or block diagrams of processes, devices (systems), and computer program products according to embodiments of the invention. It is pointed out that each block of the flowchart representations and / or block diagrams, as well as combinations of blocks in the flowchart representations and / or block diagrams, can be executed by means of computer-readable program instructions.

[0064] These computer-readable program instructions can be provided to a processor of a computer or other programmable data processing device to create a machine such that the instructions executed by the processor of the computer or other programmable data processing device produce a means of implementing the functions / steps specified in the block(s) of the flowcharts and / or block diagrams. These computer-readable program instructions can also be stored on a computer-readable storage medium capable of controlling a computer, programmable data processing device, and / or other units to function in a certain manner, such that the computer-readable storage medium on which instructions are stored has a manufactured product, including instructions specifying which aspects of the block(s) in the flowchart(s) are to be implemented.Implement the function / step specified in the blocks of the flowchart and / or block diagrams.

[0065] The computer-readable program instructions can also be loaded onto a computer, other programmable data processing device, or other unit to cause a series of process steps to be executed on the computer or other programmable device or other unit in order to produce a process executed on a computer, such that the instructions executed on the computer, other programmable device, or other unit implement the functions / steps specified in the block(s) of the flowchart and / or block diagram.

[0066] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, processes, and computer program products according to various embodiments of the present invention. In this context, each block in the flowcharts or block diagrams can represent a module, segment, or part of instructions that includes one or more executable instructions for performing the specific logical function(s). In some alternative embodiments, the functions specified in the blocks may occur in a different order than shown in the figures. For example, two blocks shown consecutively may in reality be executed simultaneously, substantially simultaneously, partially, or completely overlapping in time in one step, or the blocks may sometimes be executed in reverse order depending on the corresponding functionality.It should also be noted that each block of the block diagrams and / or flowcharts, as well as combinations of blocks in the block diagrams and / or flowcharts, can be implemented by special hardware-based systems that perform the specified functions or steps, or execute combinations of special hardware and computer instructions.

[0067] The descriptions of the various embodiments are provided for illustrative purposes and are neither exhaustive nor limited to the disclosed embodiments. Many modifications and variations are recognizable to the person skilled in the art without deviating from the scope of the described embodiments. The terminology used here was chosen to explain the principles of the embodiments, their practical application, or the technical improvements compared to technologies available on the market as clearly as possible, or to enable the person skilled in the art to understand the embodiments disclosed herein.

Claims

[1] A method for searching for images, the method comprising: Identifying a reference object in a two-dimensional reference image; Determining a three-dimensional reference alignment axis of the reference object based on at least one attribute of the reference object; Receiving an input requesting a change in a three-dimensional perspective of the reference object; Determine a new alignment axis based on the input and the reference alignment axis; Performing a search query on a set of two-dimensional images, the search query being based on the new alignment axis and at least one attribute of the reference object; and Display image search results ranked based on correlations with the new alignment axis and at least one attribute of the reference object. [2] The method of claim 1, wherein the input comprises information about a rotation gesture performed by a user. [3] The method of claim 2, wherein the information includes a direction of the rotation gesture and an angular movement of the rotation gesture. [4] The method of any preceding claim, further comprising sending a request to an external image provider resource if the image search result does not return any results to provide images corresponding to the new alignment axis and the at least one attribute of the reference object. [5] The method of claim 4, wherein the method further comprises: Receiving an image from the external image provider resource; identifying a received image object in the received image; Determining a reference alignment axis in the received image for the received image object; and if the reference alignment axis in the received image and at least one attribute of the object in the received image match the reference alignment axis and the at least one attribute of the reference image to a predetermined degree, adding the received image to the set of two-dimensional images. [6] Method according to one of claims 4 or 5, wherein the external image provider resource is a crowdsourcing source. [7] A method according to any one of the preceding claims, wherein the reference alignment axis is an axis passing through a calculated center of mass of the reference object and extending from a forward-facing side of the reference object to a rearward-facing side of the reference object. [8] A method according to any one of the preceding claims, wherein the method further comprises: Displaying a three-dimensional perspective of a sphere to a user; and Updating a rotation view of the sphere to track a user input gesture. [9] The method of claim 8, wherein the sphere has a plurality of surface portions corresponding to a plurality of different view angles of the reference object, and the surface portions have visual attributes indicating whether a corresponding image of the reference object is present in the set of two-dimensional images for the corresponding view angle. [10] The method of claim 9, wherein the visual attributes of the surface portions further indicate a degree of correspondence between the associated images in the set of images and the reference image. [11] A computer system for searching images, the system comprising: Computer-readable storage medium with program instructions stored thereon; and one or more processors configured to execute the program instructions to perform a method comprising: Identifying a reference object in a two-dimensional reference image; Determining a three-dimensional reference alignment axis of the reference object based on at least one attribute of the reference object; Receiving an input requesting a change in a three-dimensional perspective of the reference object; Determine a new alignment axis based on the input and the reference alignment axis; Performing a search query on a set of two-dimensional images, the search query being based on the new alignment axis and at least one attribute of the reference object; and Display image search results ranked based on correlations with the new alignment axis and at least one attribute of the reference object. [12] The computer system of claim 11, wherein the input comprises information about a rotation gesture performed by a user. [13] The computer system of claim 12, wherein the information includes a direction of the rotation gesture and an angular movement of the rotation gesture. [14] The computer system of any of claims 11 to 13, wherein the method further comprises sending a request to an external image provider resource if the image search result does not return any results to provide images corresponding to the new alignment axis and the at least one attribute of the reference object. [15] The computer system of claim 14, wherein the method further comprises: Receiving an image from the external image provider resource; identifying a received image object in the received image; Determining a reference alignment axis in the received image for the received image object; and if the reference alignment axis in the received image and at least one attribute of the object in the received image match the reference alignment axis and at least one attribute of the reference image to a predetermined degree, adding the received image to the set of two-dimensional images. [16] A computer program product for searching for images, the computer program product comprising: A computer-readable storage medium readable by a processing circuit and storing instructions for execution by the processing circuit to perform a method according to any one of claims 1 to 10. [17] A computer program stored on a computer-readable medium and loadable into the internal memory of a digital computer, and comprising software code portions when the program is executed on a computer to perform the method of any one of claims 1 to 10.