Image processing method, apparatus and system
Through image acquisition and SLAM and AR technology based on volumetric data, the problem of high cost of existing three-dimensional modeling and long training time of NeRF schemes is solved, and the three-dimensional information presentation with low cost and low data acquisition is realized, suitable for 360-degree surrounding scenarios .
Patent Information
- Application Number
- PCT/CN2024/089111
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-11
- Filing Date
- 2024-04-22
- Publication Date
- 2024-11-14
AI Technical Summary
The existing three-dimensional modeling technology requires high-end equipment, high-quality environment and high cost. It fails to reconstruct items with high reflectivity, transparency and fewer textures, which cannot meet market demand. The training of the NeRF scheme takes a long time and the PSNR is not high, and the synthetic pictures exist. Cloud-like noise.
Using an image acquisition method based on volume data, a normal image acquisition device is used to collect data at a designated location, and the image converted into multiple viewing angles is reduced by processing, combined with SLAM, AR and other technologies, the cost of presenting three-dimensional information is reduced.
It realizes the low-cost and low-data acquisition of three-dimensional information presentation, reduces the cost of three-dimensional information presentation, improves the presentation effect, and is suitable for 360-degree surrounding scenes.
Smart Images

Figure CN2024089111_14112024_PF_FP_ABST
Abstract
Description
Image processing method, device and system
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application is based on the application with CN application number 202310532786.0 and application date May 11, 2023, and claims its priority. The disclosed content of the CN application is hereby introduced as a whole into this application. Technical Field
[0003] The present disclosure relates to the field of image processing technology, and in particular to an image processing method, an image processing device, an image processing system, and a non-volatile computer-readable storage medium. Background Art
[0004] With the development of AR (Augmented Reality) and VR (Virtual Reality) technologies, the demand for browsing online information is no longer limited to two-dimensional information, but now requires more three-dimensional information. Therefore, the demand for three-dimensional information presentation of objects is increasing.
[0005] In related technologies, the presentation of three-dimensional information mainly relies on three-dimensional reconstruction technology, that is, presenting the three-dimensional information of an object by constructing a three-dimensional model of the object.
[0006] Summary of the Invention
[0007] According to some embodiments of the present disclosure, a method for processing images on the image acquisition side is provided, including: determining a three-dimensional area based on the position of a captured object, wherein the captured object is located in whole or in part inside the three-dimensional area; setting a plurality of image capture positions on the surface of the three-dimensional area; and using an image capture device to capture, at each image capture position, a first image of the captured object at each viewing angle of the image capture position, so as to obtain a plurality of first images of the captured object at different viewing angles, so that an image display device displays an image corresponding to the selected viewing angle based on the plurality of first images.
[0008] In some embodiments, each of the plurality of image capturing positions is equidistant from the captured object.
[0009] In some embodiments, the three-dimensional area includes a hemispherical area, and the collected object includes the center of the bottom of the hemispherical area.
[0010] In some embodiments, setting a plurality of image acquisition positions on the surface of the three-dimensional area includes: starting from an initial position of the image acquisition device, and setting an image acquisition position at every preset longitude or preset latitude.
[0011] In some embodiments, setting a plurality of image acquisition positions on the surface of the three-dimensional area includes: starting from an initial position of the image acquisition device and setting an image acquisition position at a preset distance.
[0012] In some embodiments, the image processing method further includes: determining the posture of the image acquisition device when capturing the first image at each image acquisition position as a first posture; adjusting the first posture to determine a second posture of the image acquisition device at each image acquisition position, the second posture making the y-axis of the body coordinate system of the image acquisition device at each image acquisition position point to the same point in a direction away from the bottom surface of the three-dimensional area, and the z-axis point to the captured object; performing a transmission conversion on the first image according to the first posture and the second posture to determine a second image, so as to display an image corresponding to the selected viewing angle on the image display device according to the second image.
[0013] In some embodiments, the image processing method also includes: providing relevant information of multiple first images to the image display device, so that the image display device generates an image to be displayed based on the relevant information of the first image at the specified image acquisition position and the relevant information of the first image at the adjacent image acquisition position of the specified image acquisition position, and the specified image acquisition position is determined according to the selected viewing angle.
[0014] In some embodiments, capturing a first image of the captured object at a viewing angle at each image capture position includes: capturing multiple first images at multiple image capture positions according to a rendering identifier, and the rendering identifier is kept in the same plane as the captured object through plane recognition processing.
[0015] According to other embodiments of the present disclosure, a method for processing an image on an image display side is provided, comprising: obtaining a viewing angle selected by a user; and displaying an image corresponding to the selected viewing angle based on a plurality of first images of a captured object corresponding to different viewing angles captured at a plurality of image capture positions, wherein the plurality of image capture positions are located on a surface of a three-dimensional area, the three-dimensional area is determined according to the position of the captured object, and the captured object is located in whole or in part inside the three-dimensional area.
[0016] In some embodiments, displaying an image corresponding to a selected viewing angle includes generating an image to be displayed based on relevant information of a first image at a designated image acquisition position and relevant information of a first image at an adjacent image acquisition position of the designated image acquisition position, wherein the designated image acquisition position is determined based on the selected viewing angle.
[0017] According to some further embodiments of the present disclosure, an image processing system is provided, comprising: an image acquisition device for executing the image processing method on the image acquisition side in any of the above embodiments; and an image display device for executing the image processing method on the image display side in any of the above embodiments based on a plurality of first images provided by a first processing device.
[0018] In some embodiments, the image processing system further includes: a server device configured to receive a plurality of first images uploaded by the image acquisition device and to send relevant information of the plurality of first images to the image display device.
[0019] In some embodiments, the server device sends the second image to the image display device; the image display device displays an image corresponding to the selected viewing angle based on the second image.
[0020] According to some further embodiments of the present disclosure, an image processing device is provided, comprising: a determining unit for determining a three-dimensional area based on the position of a captured object, wherein the captured object is located in whole or in part inside the three-dimensional area; a setting unit for setting a plurality of image capture positions on the surface of the three-dimensional area; and a capturing unit for capturing, at each image capture position, a first image of the captured object at a viewing angle of the captured object, using an image capturing device, to obtain a plurality of first images of the captured object at different viewing angles, so that an image display device displays an image corresponding to the selected viewing angle based on the plurality of first images.
[0021] In some embodiments, each of the plurality of image capturing positions is equidistant from the captured object.
[0022] In some embodiments, the three-dimensional area includes a hemispherical area, and the collected object includes the center of the bottom of the hemispherical area.
[0023] In some embodiments, the setting unit takes the initial position of the image acquisition device as a starting point and sets an image acquisition position at every preset longitude or preset latitude.
[0024] In some embodiments, the setting unit sets an image acquisition position at a preset distance starting from the initial position of the image acquisition device.
[0025] In some embodiments, the determination unit determines the posture of the image acquisition device when capturing the first image at each image acquisition position as a first posture; adjusts the first posture to determine a second posture of the image acquisition device at each image acquisition position, the second posture making the y-axis of the body coordinate system of the image acquisition device at each image acquisition position point to the same point in a direction away from the bottom surface of the three-dimensional area, and the z-axis points to the captured object; performs transmission conversion on the first image according to the first posture and the second posture to determine the second image, so as to display an image corresponding to the selected viewing angle on the image display device according to the second image.
[0026] In some embodiments, the image processing device also includes: a sending unit for providing relevant information of multiple first images to the image display device, so that the image display device generates an image to be displayed based on the relevant information of the first image at the specified image acquisition position and the relevant information of the first image at the adjacent image acquisition position of the specified image acquisition position, and the specified image acquisition position is determined according to the selected viewing angle.
[0027] In some embodiments, the acquisition unit acquires multiple first images at multiple image acquisition positions according to the rendering identifier, and the rendering identifier is kept in the same plane as the acquired object through plane recognition processing.
[0028] According to some further embodiments of the present disclosure, there is provided an image processing device, comprising: an acquisition unit for acquiring a viewing angle selected by a user; a display unit for displaying an image corresponding to the selected viewing angle based on a plurality of first images of a captured object corresponding to different viewing angles captured at a plurality of image capture positions, wherein the plurality of image capture positions are located on a surface of a three-dimensional area, the three-dimensional area is determined according to the position of the captured object, and the captured object is located in whole or in part inside the three-dimensional area.
[0029] In some embodiments, the display unit generates an image to be displayed based on relevant information of the first image at a designated image acquisition position and relevant information of the first image at an image acquisition position adjacent to the designated image acquisition position, where the designated image acquisition position is determined according to the selected viewing angle.
[0030] According to some further embodiments of the present disclosure, there is provided an image processing device, comprising: a memory; and a processor coupled to the memory, wherein the processor is configured to execute the image processing method in any one of the above embodiments based on instructions stored in the memory device.
[0031] According to some further embodiments of the present disclosure, a non-volatile computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the image processing method in any of the above embodiments is implemented.
[0032] According to some further embodiments of the present disclosure, a computer program is provided, comprising: instructions, which, when executed by a processor, cause the processor to execute the image processing method in any one of the above embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.
[0034] The present disclosure can be more clearly understood from the following detailed description with reference to the accompanying drawings:
[0035] FIG1 is a flowchart illustrating some embodiments of the image processing method disclosed herein;
[0036] 2a-2b are schematic diagrams showing some embodiments of the image acquisition position setting method disclosed herein;
[0037] 3a-3b are schematic diagrams showing other embodiments of the image acquisition position setting method disclosed herein;
[0038] 4a-4b are schematic diagrams showing some embodiments of the posture adjustment method disclosed herein;
[0039] 5a to 5c are schematic diagrams showing some embodiments of three-dimensional information presentation according to the present disclosure;
[0040] FIG6 is a schematic diagram showing some embodiments of an image processing system according to the present disclosure;
[0041] FIG7 shows a block diagram of some embodiments of an image processing system of the present disclosure;
[0042] 8a to 8d are block diagrams illustrating some embodiments of an image processing apparatus according to the present disclosure. DETAILED DESCRIPTION
[0043] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present disclosure.
[0044] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.
[0045] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.
[0046] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.
[0047] In all examples shown and discussed herein, any specific values should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.
[0048] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.
[0049] The inventors of the present disclosure have discovered the following problems in the above-mentioned related technologies: three-dimensional modeling of objects requires high-end acquisition equipment, a high-quality shooting environment, a long reconstruction cycle, and high manual post-processing costs, resulting in high costs for presenting three-dimensional image information.
[0050] In view of this, the present disclosure proposes an image processing technology solution that can reduce the cost of presenting three-dimensional image information.
[0051] As mentioned earlier, 3D modeling of objects requires high-end acquisition equipment, a high-quality shooting environment, a long reconstruction cycle, and high manual post-processing costs. Furthermore, 3D modeling can fail for highly reflective, transparent, or textured objects. Consequently, 3D reconstruction technology is expensive and cannot meet the growing market demand for 3D models.
[0052] The NeRF (Neural Radiance Fields) solution uses implicit representation to synthesize images of objects from different perspectives. However, NerF also has certain technical issues and limitations: NeRF model training is time-consuming, and the PSNR (Peak Signal to Noise Ratio) of the synthesized images is low. Especially for 360-degree surround scenes, the NeRF model also produces cloud-like noise in some synthesized images, which reduces the information presentation quality and fails to meet information presentation requirements.
[0053] In response to the above technical problems, the present invention discloses a method for collecting volumetric data, which uses an ordinary image acquisition device (such as a mobile phone, etc.) to collect data at a specified location; and by processing the collected data, it converts it into pictures corresponding to multiple viewing angles, thereby reducing the cost of three-dimensional information presentation at a low data collection cost.
[0054] In some embodiments, the low-cost 3D information presentation technology proposed in this disclosure integrates SLAM (Simultaneous Localization and Mapping), AR, and codec technologies to create a comprehensive 3D information presentation solution, from acquisition and data processing to display and presentation. This solution uses common image acquisition devices (such as mobile phones) to collect data; automated data management and conversion is performed on the server; and the converted data is downloaded to the player, allowing for flexible presentation of object information from various viewing angles.
[0055] FIG1 shows a flowchart of some embodiments of the image processing method disclosed herein.
[0056] As shown in FIG1 , in step 110 , a three-dimensional area is determined based on the position of the collected object, and the collected object is entirely or partially located inside the three-dimensional area.
[0057] In some embodiments, the distance between each of the plurality of image acquisition positions and the captured object is equal. For example, the three-dimensional area includes a hemispherical area (ie, a volume hemisphere), and the captured object includes the center of the bottom of the hemispherical area.
[0058] For example, the center of the bounding box of the collected item coincides with the center of the bottom of the hemispherical area.
[0059] For example, the z-axis of the coordinate system of the collected object coincides with the line from the center of the bottom surface to the vertex of the volume hemisphere.
[0060] In some embodiments, a three-dimensional area (such as a volume hemisphere) containing the captured object is determined based on the current posture of the image capture device capturing data and the size of the captured object (such as the size of the bounding box of the object).
[0061] This ensures that the camera positions corresponding to each image are located on the same hemisphere when the image is presented (i.e., the distance between each image acquisition position and the captured object is equal), thus resolving the technical issue of poor 3D information presentation caused by variations in camera position, such as perspective and rotation, during acquisition. The 3D region setting ensures that the size of objects in each final image is essentially consistent, and that the final rotation of the image acquisition follows a fixed distribution pattern, ensuring that the 3D information presented resembles a 3D model.
[0062] In step 120 , a plurality of image acquisition positions are set on the surface of the three-dimensional area.
[0063] In some embodiments, starting from the initial position of the image acquisition device, an image acquisition position is set at every preset longitude or latitude. For example, this technical solution can be implemented through the embodiments shown in Figures 2a and 2b.
[0064] 2a-2b are schematic diagrams showing some embodiments of the image acquisition position setting method disclosed herein.
[0065] As shown in Figure 2a, image acquisition positions are set at regular intervals along the latitude and longitude directions on the volumetric hemisphere. Figure 2b is a top view of Figure 2a. In this way, the set image acquisition positions can more comprehensively cover different viewing angles, thereby improving the presentation of three-dimensional information.
[0066] In some embodiments, starting from the initial position of the image acquisition device, an image acquisition position is set at every preset distance. For example, this technical solution can be implemented through the embodiments shown in Figures 3a and 3b.
[0067] 3a-3b are schematic diagrams showing other embodiments of the image acquisition position setting method disclosed herein.
[0068] As shown in Figure 3a, a certain number of image acquisition positions are evenly spaced across the volume hemisphere, with equal distances between them. Figure 3b is a top view of Figure 3a. This allows for the acquisition of the most data possible using the fewest possible image acquisition positions.
[0069] In step 130, an image acquisition device is used to capture a first image of the captured object at each viewing angle at each image acquisition position, so as to obtain multiple first images of the captured object at different viewing angles, so that the image display device can display an image corresponding to the selected viewing angle based on the multiple first images.
[0070] In the above embodiment, by setting image capture locations on the surface of the three-dimensional area containing the object, two-dimensional images of the object at different viewing angles can be captured. By displaying the two-dimensional image corresponding to the selected viewing angle, three-dimensional information presentation of the object is achieved. This can reduce the cost of presenting three-dimensional image information.
[0071] In some embodiments, a plurality of first images are captured at a plurality of image capture positions according to the rendering mark, and the rendering mark is kept in the same plane as the captured object through plane recognition processing.
[0072] For example, rendering markers (such as bounding boxes, volumetric spheres, etc.) can be used to guide users to capture images at designated image capture locations. This can be combined with plane recognition to ensure that the rendered markers and the object being photographed (captured items) are in the same plane.
[0073] For example, the acquisition end can automatically adjust the focus according to the position of the captured object in the picture to ensure that the captured object is clear in the picture.
[0074] For example, rendering markers such as bounding boxes can be used to automatically detect whether the captured object exceeds the display range of the screen; the acquisition end can calculate whether the camera is close to the specified image acquisition position based on the current position of the camera; when the acquisition requirements (such as the captured object does not exceed the display range, the camera is located at the specified image acquisition position, etc.) are met, the acquisition end can automatically trigger the storage of the acquisition information to save the current acquisition information.
[0075] In some embodiments, after image acquisition is completed, the acquisition end can automatically package and transmit the acquired first image and its corresponding first pose, camera intrinsic parameters, bounding box and other related information to the server.
[0076] For example, at the acquisition end, the Slam system on the mobile end is used to obtain the real-time posture of the camera (image acquisition device) and then send the posture to the server.
[0077] For example, the server receives the data packet uploaded by the acquisition end, stores and manages the first image according to the number of the collected item; the server parses the collected original data (the first image and its corresponding first pose, etc.).
[0078] In some embodiments, the posture of the image acquisition device when capturing the first image at each image acquisition position is determined as the first posture; the first posture is adjusted to determine the second posture of the image acquisition device at each image acquisition position, the second posture making the y-axis of the body coordinate system of the image acquisition device at each image acquisition position point to the same point in the direction away from the bottom surface of the three-dimensional area, and the z-axis points to the captured object; according to the first posture and the second posture, the first image is subjected to transmission conversion to determine the second image, so that the image corresponding to the selected viewing angle is displayed on the image display device according to the second image.
[0079] For example, the determination of the first pose and the second pose, and the conversion process between the first image and the second image can be completed by the image acquisition end, or the image acquisition end can upload the first pose and the first image to the server end and complete it.
[0080] For example, since the original first pose (such as that collected by SLAM) has deviations and cannot guarantee global consistency, the first pose can be optimized through SfM (Structure from Motion) technology; then, the optimized first pose is converted into a second pose (which can be called a volumetric pose).
[0081] 4a-4b are schematic diagrams showing some embodiments of the posture adjustment method disclosed herein.
[0082] As shown in FIG4 a , the direction of the triangle in the figure represents the first pose when the first image is acquired, and there are deviations in distance and angle between each first pose.
[0083] However, when displaying the images corresponding to each image acquisition position, the above-mentioned posture deviation is not introduced, which leads to inconsistent directions of the captured objects in the images corresponding to different image acquisition positions. For example, the tops of the captured objects in some images are facing upwards, while the bottoms of the captured objects in some images are facing upwards, resulting in errors in the final three-dimensional information presentation.
[0084] As shown in Figure 4b, the orientation of the triangle in the figure represents the adjusted second posture. This second posture ensures that the y-axis of the body coordinate system of the image acquisition device at each image acquisition position is uniformly pointed away from the bottom surface of the three-dimensional area and toward the same point (which can be a point on the z-axis of the body coordinate system of the object), and the z-axis uniformly points toward the captured object.
[0085] According to the relationship between the first posture and the second posture corresponding to the same image sampling position, the transformation matrix between the images corresponding to the two postures can be calculated; the first image is subjected to transmission transformation processing using the transformation matrix to obtain the second image.
[0086] Each second image can ensure that the collected object it presents is located at the center of the image; in response to the user adjusting the viewing angle, by switching to display the second image corresponding to the viewing angle, the three-dimensional information of the collected object can be rotated and presented, and the effect is like rotating a real three-dimensional model.
[0087] In some embodiments, the second image may be saliency segmented to eliminate the background portion of the second image; the processed second image may be encoded and compressed and then uploaded to the server; and the data may be sent when the player requests the data.
[0088] 5a to 5c are schematic diagrams showing some embodiments of three-dimensional information presentation according to the present disclosure.
[0089] As shown in FIG5a to FIG5c, different images of the captured object are presented in response to different viewing angles selected by the user.
[0090] In some embodiments, relevant information of multiple first images is provided to an image display device so that the image display device generates an image to be displayed based on relevant information of the first image at a specified image acquisition position and relevant information of the first image at an adjacent image acquisition position of the specified image acquisition position, where the specified image acquisition position is determined based on a selected viewing angle.
[0091] For example, the playback end requests the data packet corresponding to the object from the server end and downloads the data packet; identifies the viewing angle corresponding to the user's operation on the screen (such as the sliding direction, distance or zoom), and converts the viewing angle into an image acquisition position; decodes the second image of the image acquisition position and its nearby image acquisition positions (such as adjacent image acquisition positions) in the data packet; synthesizes these second images into an image of the image acquisition position that the user needs to display, and displays it on the screen, thereby realizing the three-dimensional information presentation of the object.
[0092] In this way, synthesizing images from multiple image acquisition positions can solve technical problems such as blur, light and dark changes, and posture deviation that occur during the image acquisition process and cause poor presentation effects.
[0093] FIG6 shows a schematic diagram of some embodiments of an image processing system according to the present disclosure.
[0094] As shown in Figure 6, on the capture end, a mobile Slam system such as ARKit is used to obtain the real-time position and pose of the camera (image acquisition device). Rendering markers (such as bounding boxes and volumetric spheres) guide the user to capture images at designated capture locations. Plane recognition can be combined to ensure that the rendered markers and the object being photographed (captured items) are in the same plane.
[0095] Image acquisition locations can be set up in two ways: by latitude and longitude and uniformly. Latitude and longitude acquisition involves setting an image acquisition location at regular intervals along the latitude and longitude of the volume hemisphere. Uniform acquisition involves setting a certain number of image acquisition locations evenly spaced across the volume hemisphere. This allows for maximum data capture using a minimum number of image acquisition locations.
[0096] The acquisition end can automatically adjust the focus according to the position of the captured object in the picture to ensure that the captured object is clear in the picture.
[0097] Rendering marks such as bounding boxes can be used to automatically detect whether the captured object exceeds the display range of the screen; the capture end can calculate whether the camera is close to the specified image capture position based on the current position of the camera; when the capture requirements (such as the captured object does not exceed the display range, the camera is located at the specified image capture position, etc.) are met, the capture end can automatically trigger the storage of the capture information to save the current capture information.
[0098] After the image acquisition is completed, the acquisition end can automatically package and transmit the collected first image and its corresponding first pose, camera intrinsic parameters, bounding box and other related information to the server.
[0099] The server receives the data packet uploaded by the acquisition end, stores and manages the first image according to the number of the collected item; the server parses the collected original data (the first image and its corresponding first pose, etc.).
[0100] Since the original first pose (such as that collected by SLAM) has deviations and cannot guarantee global consistency, the first pose can be optimized through SfM technology; then, the optimized first pose is converted into the second pose.
[0101] The second image can be saliency segmented to eliminate the background portion of the second image; the processed second image is encoded and compressed and then uploaded to the server; when the player requests data, the data is sent down.
[0102] The playback end requests the data packet corresponding to the object from the server and downloads the data packet; identifies the viewing angle corresponding to the user's operation on the screen (such as the direction, distance or zoom of sliding), and converts the viewing angle into an image acquisition position; decodes the second image of the image acquisition position and its nearby (i.e., within a preset range) image acquisition positions in the data packet; synthesizes these second images into an image of the image acquisition position that the user needs to display, and displays it on the screen, thereby realizing the three-dimensional information presentation of the object.
[0103] FIG7 shows a block diagram of some embodiments of an image processing system of the present disclosure.
[0104] As shown in Figure 7, the image processing system 7 includes: an image acquisition device 71, which is used to execute the image processing method on the image acquisition side in any of the above-mentioned embodiments; and an image display device 73, which is used to execute the image processing method on the image display side in any of the above-mentioned embodiments based on multiple first images provided by the first processing device.
[0105] In some embodiments, the image processing system 7 further includes a server device 72 for receiving a plurality of first images uploaded by the image acquisition device and sending relevant information of the plurality of first images to the image display device.
[0106] In some embodiments, the server device 72 sends the second image to the image display device 73; the image display device 73 displays an image corresponding to the selected viewing angle based on the second image.
[0107] 8a to 8d are block diagrams illustrating some embodiments of an image processing apparatus according to the present disclosure.
[0108] As shown in Figure 8a, the image processing device 8a includes: a determination unit 81a, used to determine a three-dimensional area based on the position of the collected object, and the collected object is located in whole or in part inside the three-dimensional area; a setting unit 82a, used to set multiple image collection positions on the surface of the three-dimensional area; an acquisition unit 83a, used to use the image acquisition device to collect, at each image collection position, a first image of the collected object at each viewing angle, so as to obtain multiple first images of the collected object at different viewing angles, so that the image display device can display the image corresponding to the selected viewing angle based on the multiple first images.
[0109] In some embodiments, each of the plurality of image capturing positions is equidistant from the captured object.
[0110] In some embodiments, the three-dimensional area includes a hemispherical area, and the collected object includes the center of the bottom of the hemispherical area.
[0111] In some embodiments, the setting unit 82a takes the initial position of the image acquisition device as a starting point and sets an image acquisition position at every preset longitude or latitude.
[0112] In some embodiments, the setting unit 82a takes the initial position of the image acquisition device as a starting point and sets an image acquisition position at every preset distance.
[0113] In some embodiments, the determination unit 81a determines the posture of the image acquisition device when capturing the first image at each image acquisition position as the first posture; adjusts the first posture to determine the second posture of the image acquisition device at each image acquisition position, the second posture making the y-axis of the body coordinate system of the image acquisition device at each image acquisition position point to the same point in the direction away from the bottom surface of the three-dimensional area, and the z-axis points to the captured object; performs transmission conversion on the first image according to the first posture and the second posture to determine the second image, so as to display the image corresponding to the selected viewing angle on the image display device according to the second image.
[0114] In some embodiments, the image processing device 8a also includes: a sending unit 84a, used to provide relevant information of multiple first images to the image display device, so that the image display device generates an image to be displayed based on the relevant information of the first image at the specified image acquisition position and the relevant information of the first image at the adjacent image acquisition position of the specified image acquisition position, and the specified image acquisition position is determined according to the selected viewing angle.
[0115] In some embodiments, the acquisition unit 83a acquires multiple first images at multiple image acquisition positions according to the rendering mark, and the rendering mark is kept in the same plane as the captured object through plane recognition processing.
[0116] As shown in FIG8b , the image processing device 8b includes: an acquisition unit 81b for acquiring a viewing angle selected by a user; and a display unit 82b for displaying an image corresponding to the selected viewing angle based on a plurality of first images of the captured object corresponding to different viewing angles captured at a plurality of image capture positions, wherein the plurality of image capture positions are located on the surface of a three-dimensional area, the three-dimensional area is determined according to the position of the captured object, and the captured object is located in whole or in part inside the three-dimensional area.
[0117] In some embodiments, the display unit 82b generates an image to be displayed based on relevant information of the first image at a designated image acquisition position and relevant information of the first image at an adjacent image acquisition position of the designated image acquisition position, where the designated image acquisition position is determined based on the selected viewing angle.
[0118] As shown in Figure 8c, the image processing device 8c of this embodiment includes: a memory 81c and a processor 82c coupled to the memory 81c, and the processor 82c is configured to execute the image processing method in any embodiment of the present disclosure based on the instructions stored in the memory 81c.
[0119] The memory 81c may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, application programs, a boot loader, a database, and other programs.
[0120] As shown in FIG8 d , the image processing device 8d of this embodiment includes a memory 81d and a processor 82d coupled to the memory 81d . The processor 82d is configured to execute the image processing method of any of the aforementioned embodiments based on instructions stored in the memory 81d .
[0121] The memory 81d may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, application programs, a boot loader, and other programs.
[0122] The image processing device 8d may also include an input / output interface 83d, a network interface 84d, a storage interface 85d, and the like. These interfaces 83d, 84d, and 85d, as well as the memory 81d and the processor 82d, may be connected, for example, via a bus 86d. The input / output interface 83d provides a connection interface for input / output devices such as a display, mouse, keyboard, touch screen, microphone, and speakers. The network interface 84d provides a connection interface for various networked devices. The storage interface 85d provides a connection interface for external storage devices such as SD cards and USB flash drives.
[0123] Those skilled in the art will appreciate that embodiments of the present disclosure may be provided as methods, systems, or computer program products. Thus, the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable non-transitory storage media, including but not limited to magnetic disk storage, CD-ROMs, optical storage, and the like, containing computer-usable program code.
[0124] The image processing method, image processing apparatus, image processing system, and non-volatile computer-readable storage medium according to the present disclosure have been described in detail. To avoid obscuring the concepts of the present disclosure, some details known in the art have been omitted. Based on the above description, those skilled in the art will fully understand how to implement the technical solutions disclosed herein.
[0125] The methods and systems of the present disclosure may be implemented in many ways. For example, the methods and systems of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above unless otherwise specified. In addition, in some embodiments, the present disclosure may also be implemented as programs recorded in a recording medium, which include machine-readable instructions for implementing the methods according to the present disclosure. Therefore, the present disclosure also covers recording media that store programs for executing the methods according to the present disclosure.
[0126] Although some specific embodiments of the present disclosure have been described in detail by way of examples, those skilled in the art will appreciate that the above examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Those skilled in the art will appreciate that modifications may be made to the above embodiments without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.
Claims
1. A method for processing an image, comprising: Determine a three-dimensional area according to the position of the collected object, wherein the collected object is located entirely or partially inside the three-dimensional area; On the surface of the three-dimensional area, a plurality of image acquisition positions are set; By using an image acquisition device, at each image acquisition position, a first image of the captured object at the viewing angle of each image acquisition position is captured to obtain multiple first images of the captured object at different viewing angles, so that an image display device can display an image corresponding to the selected viewing angle based on the multiple first images.
2. The processing method according to claim 1, wherein: The distance between each of the plurality of image acquisition positions and the acquired object is equal.
3. The processing method according to claim 2, wherein: The three-dimensional area includes a hemispherical area, and the collected objects include the center of the bottom of the hemispherical area.
4. The processing method according to claim 3, wherein: The setting of a plurality of image acquisition positions on the surface of the three-dimensional area comprises: Taking the initial position of the image acquisition device as the starting point, an image acquisition position is set every preset longitude or preset latitude.
5. The processing method according to any one of claims 1 to 4, wherein: The setting of a plurality of image acquisition positions on the surface of the three-dimensional area comprises: Taking the initial position of the image acquisition device as the starting point, an image acquisition position is set at every preset distance.
6. The processing method according to any one of claims 1 to 5, further comprising: Determine the posture of the image acquisition device when acquiring the first image at each image acquisition position as the first posture; The first posture is adjusted to determine a second posture of the image acquisition device at each image acquisition position, The second posture enables the y-axis of the body coordinate system of the image acquisition device at each image acquisition position to point to the same point in a direction away from the bottom surface of the three-dimensional area, and the z-axis to point to the captured object; According to the first posture and the second posture, the first image is subjected to transmission conversion to determine a second image, so that an image corresponding to the selected viewing angle is displayed on the image display device according to the second image.
7. The processing method according to any one of claims 1 to 5, further comprising: Provide relevant information of the multiple first images to the image display device so that the image display device generates an image to be displayed based on relevant information of the first image at a specified image acquisition position and relevant information of the first image at an image acquisition position adjacent to the specified image acquisition position, wherein the specified image acquisition position is determined based on the selected viewing angle.
8. The processing method according to any one of claims 1 to 5, wherein: The step of acquiring a first image of the captured object at a viewing angle at each image acquisition position comprises: The plurality of first images are acquired at the plurality of image acquisition positions according to the rendering mark, and the rendering mark is kept in the same plane as the acquired object through plane recognition processing.
9. A method for processing an image, comprising: Get the viewing angle selected by the user; Based on multiple first images of the collected object corresponding to different viewing angles collected at multiple image collection positions, an image corresponding to the selected viewing angle is displayed, the multiple image collection positions are located on the surface of a three-dimensional area, the three-dimensional area is determined according to the position of the collected object, and the collected object is located in whole or in part inside the three-dimensional area.
10. The processing method according to claim 9, wherein: The displaying of the image corresponding to the selected viewing angle comprises: An image to be displayed is generated based on relevant information of a first image at a designated image acquisition position and relevant information of a first image at an image acquisition position adjacent to the designated image acquisition position, wherein the designated image acquisition position is determined based on the selected viewing angle.
11. An image processing system, comprising: An image acquisition device, used to execute the image processing method according to any one of claims 1 to 8; An image display device, used to execute the image processing method described in claim 9 or 10 based on the multiple first images provided by the first processing device.
12. The processing system of claim 11, further comprising: The server device is used to receive the multiple first images uploaded by the image acquisition device and send relevant information of the multiple first images to the image display device.
13. The processing system of claim 12, wherein: The server device sends a second image to the image display device; The image display device displays an image corresponding to the selected viewing angle based on the second image.
14. An image processing device, comprising: A determination unit, configured to determine a three-dimensional area according to the position of the collected object, wherein the collected object is wholly or partially located inside the three-dimensional area; A setting unit, used for setting a plurality of image acquisition positions on the surface of the three-dimensional area; The acquisition unit is used to use the image acquisition device to acquire, at each image acquisition position, a first image of the captured object at the viewing angle of the captured object at the image acquisition position, so as to obtain multiple first images of the captured object at different viewing angles, so that the image display device can display the image corresponding to the selected viewing angle based on the multiple first images.
15. The processing device according to claim 14, further comprising: A sending unit is used to provide the image display device with relevant information of the multiple first images so that the image display device generates an image to be displayed based on the relevant information of the first image at a specified image acquisition position and the relevant information of the first image at an image acquisition position adjacent to the specified image acquisition position, wherein the specified image acquisition position is determined based on the selected viewing angle.
16. An image processing device, comprising: An acquisition unit, used to acquire a viewing angle selected by a user; A display unit is used to display an image corresponding to the selected viewing angle based on a plurality of first images of the collected object corresponding to different viewing angles collected at a plurality of image collection positions, wherein the plurality of image collection positions are located on a surface of a three-dimensional area, the three-dimensional area is determined according to the position of the collected object, and the collected object is located in whole or in part inside the three-dimensional area.
17. An image processing device, comprising: Memory; and A processor coupled to the memory, wherein the processor is configured to execute the image processing method according to any one of claims 1 to 10 based on instructions stored in the memory.
18. A non-volatile computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the image processing method according to any one of claims 1 to 10 is implemented.
19. A computer program comprising: An instruction, when executed by a processor, causes the processor to execute the image processing method according to any one of claims 1 to 10.
Citation Information
Patent Citations
Panoramic imaging system of micro-dimensional objects and implementation method of panoramic imaging system
CN107452053A
Image processing method, device and system and storage medium
CN109658365A
Positioning, grabbing, data processing and display method and device for augmented reality
CN111694423A
Key frame image acquisition method, device and system and three-dimensional reconstruction method
CN112348958A
Resource object three-dimensional model generation interaction method and device and resource object three-dimensional model display method and device
CN114596407A
