Object tracking method based on 360-degree panorama

Abstract

The invention discloses an object tracking method based on a 360-degree panorama, and relates to the field of image processing, and the method comprises the steps: rendering a background and a tracking object in a home decoration design scheme, and generating the panorama of the background and the panorama of the tracking object respectively; cutting the panorama into a plurality of slices, and calculating the longitude and latitude of each slice in the sphere; sphere grids are established, the sphere grids and the slices are in one-to-one correspondence, and the corresponding slices are loaded in the sphere grids; determining the slice position of the object in the background panorama, replacing the slice in the area covered by the to-be-tracked object in the background with the panorama of the object combined background, restoring the design scheme, and loading the design scheme to a browser; and receiving an operation instruction sent by a user, and conveniently deleting, replacing and moving the to-be-tracked object in the panorama in real time according to a mode of only updating the slices in the object coverage area until the design scheme is determined. The method is convenient to operate, and tracking of a plurality of objects to be tracked in a large range is achieved with small calculated amount.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and more specifically, to a 360-degree panoramic image object tracking technology. Background Art

[0002] Existing home design software has certain space design and layout functions, but static plan designs or simple model displays cannot truly restore the on-site environment. Users often need to understand the space layout through field measurements, hand-drawing or other methods, which is not only time-consuming and laborious, but also has errors and uncertainties.

[0003] 360-degree panoramic image (hereinafter referred to as panorama) technology has shown great potential in many fields, and is also widely used in the display and browsing of design plans in the home decoration industry. In the decoration design process, users hope to be able to delete, replace and move objects in the design plan conveniently and quickly. This first requires accurate positioning of the object, and at the same time, timely calculation and feedback of the user's operation, and finally the real-time tracking of the object. Object tracking puts higher requirements on the existing 360-degree panoramic image technology. The difficulty lies in supporting users to operate the image in real time with appropriate resources and time consumption.

[0004] Existing object tracking technology usually relies on external devices such as electronic tags or sensors. The equipment has high cost and is difficult to install. It is also unable to achieve real-time and efficient tracking of multiple objects in a large space, which limits its practical application in decoration design scenarios.

[0005] The Chinese patent application, application number CN 111179025 A, published on December 13, 2019, records a roaming method of a home decoration 3D scene based on UE4, including the following steps: obtaining the user's apartment information, generating and displaying the corresponding home decoration 3D scene map through the UE4 engine; determining the user's movement route according to the user's input operation to adjust the viewing angle of the home decoration 3D scene map in real time and display it; according to the user's product replacement instruction, replacing the original product with the target product and re-rendering to generate a new home decoration 3D scene map and display it. This method is based on the UE4 engine, generates the corresponding home decoration 3D scene according to the apartment information, and uses the 3D rendering function of the UE4 engine to realize the roaming function of the home decoration 3D scene. However, this method needs to be re-rendered when the product is replaced, which consumes more resources and has weak real-time performance. In scenarios with high interaction and frequent operations, timely feedback cannot be guaranteed. Summary of the invention

[0006] 1. Technical issues to be solved

[0007] In view of the problems in the prior art of high cost, complex methods, cumbersome operations and low efficiency of object tracking equipment in panoramic images, the present invention provides an object tracking method based on panoramic images, which can track multiple objects in a large space, and can delete, replace and move objects to be tracked in the panoramic image in real time and conveniently.

[0008] 2. Technical solution

[0009] The purpose of the present invention is achieved through the following technical solutions.

[0010] The present invention provides an object tracking method based on a 360-degree panoramic image, and the method steps are as follows:

[0011] Step S100: Design a 3D home decoration design plan through 3D design software, the design plan includes decoration elements, select some or all of the decoration elements as the objects to be tracked, and mark the combination of the remaining elements as the background; the objects to be tracked are the elements in the design plan that are preset to support deletion, replacement and movement operations.

[0012] Preferably, the 3D design software is 3dsMax software.

[0013] Step S200: at least one replacement object is set for each object to be tracked in step S100, the replacement object and the object to be tracked are stored in different layers of the design scheme, and the center point of the replacement object coincides with the center point of the object to be tracked; the replacement object is a decoration element in the design scheme used to replace the original object.

[0014] Preferably, the replacement object has the same category attributes as the object to be tracked, and after the replacement object replaces the object to be tracked and is added to the background, it does not collide with other elements in the same background. The replacement object and the object to be tracked are collectively referred to as objects.

[0015] Step S300: Set the camera position, render the background, objects to be tracked and replacement objects in the home decoration design plan based on the same camera position, generate a background panorama, a panorama of the object to be tracked and a panorama of the replacement object; record the coordinates of the center point of the replacement object (Xw, Yw, Zw), the angle between the center point of the replacement object and the camera, and the scale ratio of the model.

[0016] Step S400: Load the background panorama into the 3D sphere to restore the background of the home decoration design plan; the method is: create a 3D scene through 3D design software to place all 3D objects; create a camera and set the camera position; create a renderer to render the scene in the browser; create a sphere, load the background panorama into the sphere, and restore the background of the home decoration design plan; before loading the background panorama into the sphere, cut the background panorama into slices of the same size, and establish a slice index (x ) for the slice located at row x and column y tile ,y tile ); save all slice textures obtained by cutting; divide the sphere into multiple grids, and use longitude and latitude to represent the position on the sphere; the number and size of the sphere grids are consistent with the number and size of slices divided from the background panorama; when specifying the longitude and latitude of the sphere grid, first convert the longitude and latitude into a slice index, and obtain the path where the corresponding slice texture is located through the slice index, and replace the texture at the corresponding position on the sphere with the corresponding slice texture by replacing the function of the specified slice. When replacing, convert the coordinates of the center point of the slice into longitude and latitude to accurately locate the slice to the sphere grid; traverse the sphere grid and replace the grid textures one by one with the texture of the corresponding slice in the background panorama until all grids are replaced to achieve background restoration;

[0017] Preferably, the slice is rectangular, and the width tile_width and the height tile_height of the slice are powers of 2.

[0018] Preferably, the slice is a square with a side length of T L is 256 pixels.

[0019] Furthermore, the total number of slice levels L of the panoramic image is determined according to the slice size. The calculation method is: the size of the panoramic image is divided by the slice size to get the number of slices 2 L ×2 L , further calculation to obtain the total number of levels L. Correspondingly, x tile and tile The value range is 0 to 2 L Preferably, L is 7 or 8.

[0020] Furthermore, the position coordinates of the slice in the background panorama are determined. The panorama coordinates are pixel coordinates on the screen, ranging from (0, 0). If the slice is a rectangle, the slice (x tile ,y tile ) ranges from (x tile ×tile_height,y tile ×tile_width) to ((x tile +1)×tile_height, (y tile+1)×tile_width), where tile_width is the width of the slice and tile_height is the height of the slice.

[0021] The coordinates of the slice center point in the background panorama are defined as (u, v).

[0022] Furthermore, there is a one-to-one conversion relationship between the background panorama coordinates and the longitude and latitude of the sphere. The longitude and latitude of the center point of the slice on the sphere can be calculated through the coordinates (u, v) of the center point of the slice. The calculation method is:

[0023]

[0024] Among them, (u, v) are the position coordinates of the center point of the slice in the background panorama; u is the two-dimensional horizontal coordinate of the slice in the background panorama, and v is the two-dimensional vertical coordinate of the slice in the background panorama; lon is the longitude of the slice in the sphere; lat is the latitude of the slice in the sphere; w is the width of the background panorama; h is the height of the background panorama.

[0025] The method of converting spherical longitude and latitude into background panorama slice index is:

[0026]

[0027] Among them, (x tile ,y tile ) is the slice index in the background panorama, lon is the longitude of the slice in the sphere; lat is the latitude of the slice in the sphere, and L is the total number of cut levels of the background panorama.

[0028] S400 specifically includes the following sub-steps:

[0029] Sub-step S410, cutting the background panorama: cutting the background panorama into slices of the same size, the slice index of the xth row and yth column is (x tile ,y tile ).

[0030] Sub-step S420: creating a new 3D scene object for placing all 3D objects.

[0031] Sub-step S430, creating a camera and setting the camera position. Sub-step S440, creating a renderer: setting the renderer size and attaching the renderer to the document.

[0032] Sub-step S450: creating a sphere, and applying the background panoramic image to the sphere.

[0033] The specific process of applying the background panorama to the sphere is as follows:

[0034] (2) Create spherical geometry;

[0035] (2) Loading a texture atlas: Create a texture loader and use it to load a texture atlas of a specified path. This atlas contains multiple small textures for replacing different slices on the sphere.

[0036] (3) Create a material: Create a basic mesh material and use the loaded texture atlas to render the inner surface of the sphere.

[0037] (4) Create a sphere mesh: Combine the sphere geometry and material into a mesh object to form a sphere; divide the sphere into grids with the same number and size as the background panorama slices.

[0038] (5) Add the sphere to the scene: Add the created sphere to the 3D scene;

[0039] (6) Define a function to replace a specified slice: Define a function replaceSlice to replace the texture of a specified slice with the texture of the spherical mesh. This function accepts two parameters: the slice index and the new texture path. The new texture path stores the slice texture corresponding to the slice index.

[0040] Preferably, THREE.TextureLoader of threerjs software is used to load the panoramic image.

[0041] Step S500: merge all the objects to be tracked into the background panorama, and update the texture of the grid corresponding to the coverage area of ​​the object to be tracked on the sphere; the method is: first, according to the three-dimensional coordinates of the object to be tracked and the perspective information and the background panorama Figure 2 The mapping relationship between the specific positions on the 3D plane is used to calculate the index of a group of slices within the coverage range of the outline of the object to be tracked and the displacement of the object to be tracked within the slice, and the slice position of the object to be tracked in the background panorama is obtained. Then, according to the slice position, the panorama of the object to be tracked is merged into the background panorama; finally, the merged panorama is sliced ​​according to the same rules as the background panorama to obtain a group of new slices within the coverage range of the object to be tracked, which are used to update the texture of the grid in the corresponding area of ​​the sphere; the above steps are repeated until the restoration of all objects to be tracked is completed.

[0042] Step S500 specifically includes the following sub-steps:

[0043] In sub-step S510, the three-dimensional world coordinates (Xw, Yw, Zw) of the center point of the object to be tracked are converted into coordinates in the camera coordinate system by the following formula:

[0044] Camera_Coord=View_Matrix×World_Coord

[0045] World_Coord is the coordinate vector (Xw, Yw, Zw) of the object in the world coordinate system, View_Matrix is ​​the view matrix of the camera, and Camera_Coord is the coordinate of the object to be tracked in the camera coordinate system.

[0046] Then, the coordinates of the object to be tracked in the camera coordinate system are converted into two-dimensional view coordinates under the camera's perspective through the projection matrix. The conversion formula is as follows:

[0047] Clip_Coord=Projection_Matrix×Camera_Coord

[0048] Projection_Matrix is ​​the projection matrix of the camera, and Clip_Coord is the two-dimensional perspective coordinates of the object to be tracked from the perspective of the camera.

[0049] Sub-step S520, converting the two-dimensional view coordinates Clip_Coord of the object to be tracked under the camera view into the pixel coordinates of the object to be tracked on the screen, the calculation method is:

[0050]

[0051] Among them, (Clip_Coord x , Clip_Coord y ) is the two-dimensional viewing coordinate of the object under the camera’s viewing angle; (x norm ,y norm ) is the normalized screen coordinate, ranging from -1 to 1; Clip_Coord w are the homogeneous coordinate components used for perspective division;

[0052] (x screen ,y screen ) is the pixel coordinate of the object on the screen; Screen_Width is the width of the screen, and Screen_Height is the height of the screen, both in pixels.

[0053] Sub-step S530, mapping the pixel coordinates of the object to be tracked on the screen to the slice. The position of the object to be tracked on the screen is obtained according to the coordinates of the center point of the object to be tracked, the angle between the center point of the object and camera A, and the scale of the model, and the slice position of the object in the background panorama is calculated. The slice position of the object includes two parameters: the slice index and the offset of the object in the slice.

[0054] The specific method is: Assume that the background image is divided into several slices, and the width and height of each slice are tile_width and tile_height respectively. Then the screen coordinates of the object can be mapped to the corresponding slice index of the object in the background panorama by the following formula:

[0055]

[0056] Among them, (x screen ,y screen ) is the pixel coordinate of the object on the screen; (x tile ,y tile ) is the slice index in the background panorama, tile_width is the slice width, tile_height is the slice height, and floor is the rounding operation.

[0057] Furthermore, the offset of the object to be tracked in the slice is calculated to determine the specific position of the object to be tracked relative to the slice:

[0058] x offset =x screen modtile_width

[0059] y offset =y screen modtile_height

[0060] tile_width is the width of the tile; tile_height is the height of the tile; x offset is the horizontal offset of the object in the slice; offset It is the longitudinal offset of the object in the slice; mod is the remainder operation.

[0061] Sub-step S540, according to the slice position of the object to be tracked in the background image, the panoramic image of the object to be tracked is loaded onto the background panoramic image, the slice image of the area where the object is located is regenerated, and the grid texture of the area covered by the object to be tracked on the sphere is replaced.

[0062] Step S600: Setting tracking operations for the objects to be tracked. Specifically, adding interactive events for all objects to be tracked in the scene, including deletion, replacement and move operations. The relevant events are triggered by the screen input operation obtained by the system, and the objects to be tracked are deleted, replaced or moved according to the relationship between the background panorama, the panorama of the object to be tracked and the panorama of the replacement object, and the correspondence between the panorama slices and the spherical grid.

[0063] The method for deleting the object to be tracked is: calculating all the slice indexes in the coverage area of ​​the object to be tracked in the panorama, and updating the corresponding spherical mesh texture to the corresponding slice of the background panorama according to the slice index.

[0064] Replacement operation: After the system obtains the operation of replacing the object to be tracked, it displays a group of replacement objects corresponding to the object to be tracked, obtains the information of a selected replacement object, calculates the slice area covered by the object to be tracked and the replacement object, first executes the operation of deleting the object to be tracked, and applies the method of step S500 to restore the replacement object to the background.

[0065] Operation of moving the object to be tracked: obtain the information of dragging the object to be tracked on the screen. When the target position of the object to be tracked exceeds the movable range, a reminder is given that the moving operation cannot be completed. When the target position of the object to be tracked falls within the movable range, the operation of deleting the object to be tracked is first performed, and then the method of step S500 is applied to restore the object to be tracked to the target position of the background panorama, regenerate the panoramic slice in the new area covered by the object to be tracked, and update the texture of the corresponding set of grids on the sphere.

[0066] The method to determine the movable range of the object to be tracked on the screen is:

[0067] (1) First, the preset movable range of the object to be tracked in the three-dimensional scene of the home decoration design plan is obtained, and the three-dimensional coordinates of the bounding box of the moving range of the object to be tracked are obtained.

[0068] (2) Convert the three-dimensional coordinates of the bounding box of the moving range of the object to be tracked into a movable area on the screen. The method is: use the camera's projection matrix to convert the three-dimensional coordinates of the bounding box into two-dimensional coordinates under the camera's perspective, and then further convert them into screen coordinates to obtain a rectangular area on the screen where the object to be tracked can be dragged. The minimum point of the rectangle is defined as (min_x, min_y), and the maximum point is defined as (max_x, max_y). The method for converting from three-dimensional coordinates to pixel coordinates on the screen is recorded in step S500.

[0069] Step S700: Loading a sphere obtained by merging the background panorama and the panorama of all objects to be tracked into a browser.

[0070] Step S800: Obtain the location information of the user's mouse click through the browser; determine whether the mouse click element is the object to be tracked, if not, skip it; if it is the object to be tracked, the operation options of the object to be tracked will appear: delete, replace and move, and apply the method recorded in S600 to perform the corresponding operation, and provide real-time feedback on the effect of the adjustment of the object to be tracked through slicing update; repeat the above operations until the solution modification is completed and the object tracking is completed.

[0071] After the modification of the plan is completed, the customer confirms and saves the final design plan. Furthermore, the panoramic image position corresponding to the object in the final design plan is restored to the three-dimensional coordinates through coordinate conversion, and the home decoration design plan is updated.

[0072] The present invention cuts the panorama into small, manageable slices, completes the mapping between the slice positions in the panorama and the longitude and latitude in the sphere, and in the interactive feedback process of scheme update, efficient loading and display of images in the object tracking process is achieved by updating some slices in the panorama, so that the panorama of the home improvement design scheme can be edited and adjusted in real time with visual adjustment, thereby improving the user experience.

[0073] 4. Beneficial effects

[0074] Compared with the prior art, the advantages of the present invention are: without the support of additional equipment, the deletion, replacement and movement of the objects to be tracked in the panorama can be realized, which is convenient, fast and low-cost. The panorama is generated by rendering and exporting the 3D design solution. When the panorama is restored with the same scene, lighting and camera settings, the authenticity and accuracy are higher, the user's immersive experience is stronger, and the true WYSIWYG is achieved. The background panorama, the panorama of the object to be tracked and the panorama of the replacement object are finely gridded, and the relationship between the three is established to minimize the calculation amount of the tracking operation and the amount of information transmitted over the network, reduce the time cost, increase the feedback speed, and improve the efficiency of user operations.

[0075] The present invention simplifies the tracking operation by presetting a rich amount of information about objects to be tracked and objects to be replaced, presetting the feedback process of related operation events, making the operation of related programs lightweight, greatly compressing the required resources, supporting quick loading of panoramas through browsers, mobile applets, and apps, supporting the selection of replacement objects based on object attributes such as name and type information, and easily implementing basic editing operations such as deletion, replacement, and movement of objects to be tracked, thereby realizing the object tracking operation of the present invention. Users and even junior designers can quickly and intuitively complete the adjustment, confirmation, and demonstration of home improvement design plans without any technical thresholds, and realize visual real-time updates of design plans through object tracking, which reduces communication costs and simplifies the home improvement design process. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 is a flow chart of the method steps of the present invention;

[0077] Figure 2 Background panoramas exported for rendering;

[0078] Figure 3 Export the panorama of the object to be tracked for rendering;

[0079] Figure 4 A panoramic image obtained by restoring the object to be tracked to the background;

[0080] Figure 5 A panoramic image after replacing the object to be tracked with the replacement object;

[0081] Figure 6 This is a panoramic image of the tea table, an object to be tracked in the embodiment, merged with the background;

[0082] Figure 7 This is a panoramic view after the tea table to be tracked is replaced with a new tea table in the embodiment. DETAILED DESCRIPTION

[0083] In the embodiments of the present invention, the relevant technical terms conform to the standard definitions of general home decoration design solutions. The relevant concepts appearing in the text are explained as follows:

[0084] The major categories of home decoration design include hard decoration, soft decoration, movable furniture and finished furniture, etc. The specific classification is as follows:

[0085] Hard decoration: It has fixed and immovable properties, and the replacement object must be a building structure or decorative fixture.

[0086] Soft furnishings: have the property of being easy to replace, and the replacement objects are non-fixed and changeable decorative elements.

[0087] Movable furniture: provides seating, tables or similar functions and can be moved at will.

[0088] Finished furniture: It has functions such as storage and display, usually as independent furniture units that can be replaced but still retain the same purpose and function.

[0089] This classification facilitates the implementation of industry-standard replacement designs, ensuring visual and functional consistency while meeting broad common specifications for the decor industry.

[0090] Example

[0091] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0092] like Figure 1 As shown, the present invention provides an object tracking method based on a 360-degree panoramic image, and the method steps are as follows:

[0093] Step S100: Design a 3D home decoration design scheme through 3D design software, the design scheme includes decoration elements, select some or all decoration elements as objects to be tracked, and mark the combination of the remaining elements as background; the objects to be tracked are elements in the design scheme that are preset to support deletion, replacement and movement operations. Preferably, in this embodiment, the 3D design software is 3dsMax software.

[0094] Step S200: Set at least one replacement object for each object to be tracked in step S100. There is no limit on the number of replacement objects. The replacement object and the object to be tracked are stored in different layers of the design scheme, and the center point of the replacement object coincides with the center point of the object to be tracked. The replacement object is a decoration element used to replace the original object in the design scheme.

[0095] Preferably, the replacement object has the same category attributes as the object to be tracked. After the replacement object replaces the object to be tracked and is added to the background, it does not collide with other elements in the same background. In this embodiment, the same category attributes as the replacement object and the object to be tracked include: hard furnishings, soft furnishings, movable furniture and finished furniture. In the classification of items in traditional home decoration design solutions, this embodiment limits the replacement object and the object to be tracked to be consistent in the major category. There is no restriction on the minor category attributes.

[0096] The replacement object and the object to be tracked are collectively referred to as objects.

[0097] Step S300: Render the background, the object to be tracked and the replacement object in the home decoration design scheme to generate a background panorama, a panorama of the object to be tracked and a panorama of the replacement object. Further, the generated panorama is grouped and numbered.

[0098] Specifically, S300 is divided into the following sub-steps:

[0099] Sub-step S310, rendering background: the elements marked as "background" in the design solution of step S100 are set to be displayed, and the objects to be tracked and the replacement objects are set to be hidden.

[0100] Set the appropriate camera position A, and then export the panorama through the renderer provided by the design software.

[0101] Sub-step S320, rendering the object to be tracked: Hide all the "background" and replacement objects in the home decoration design plan, display the objects to be tracked one by one in turn, and render and export each object to be tracked based on the same camera position A as sub-step S310 to export a panoramic view of the object to be tracked. At the same time, record the information of the object to be tracked in the background, including: the coordinates of the center point of the object to be tracked (Xw, Yw, Zw), the angle between the center point of the object to be tracked and the camera A, and the scale ratio of the model.

[0102] Sub-step S330, rendering replacement objects: Hide all the background and objects to be tracked in the home decoration design plan, display the replacement objects one by one in turn, render each replacement object based on the same camera position A as sub-step S310, and export the replacement object panorama. Record the coordinates (Xw, Yw, Zw) of the center point of the replacement object, the angle between the center point of the replacement object and camera A, and the scale ratio of the model.

[0103] Repeat the above steps for all backgrounds, objects to be tracked, and replacement objects to obtain a panoramic image of all backgrounds, objects to be tracked, and replacement objects.

[0104] Preferably, the panoramas of the background, the object to be tracked and the replacement object are grouped and numbered: all the background panoramas are grouped into one group, all the panoramas of the object to be tracked are grouped into one group, and all the panoramas of the replacement object are grouped into one group. The number of the panorama includes a group number part and an image number part, and objects in the same group have the same group number. Grouping and numbering the pictures can be used to search for the corresponding pictures during deletion, replacement and movement operations during object tracking, which can improve efficiency.

[0105] There is a one-to-many correspondence between the background and the object to be tracked, and a background number corresponding to the background is set in the number of the object to be tracked.

[0106] There is a one-to-many correspondence between the object to be tracked and the replacement object, and the replacement object number setting has the number of the object to be tracked.

[0107] Step S400: Create a 3D scene for placing all 3D objects; create a camera and set the camera position; create a renderer for rendering the scene in the browser; create a sphere, load the background panorama into the sphere, and complete the restoration of the background of the home decoration design plan.

[0108] Preferably, in the process of restoring the panorama, this embodiment includes a step of panorama cutting, the method is: cutting the background panorama into slices of the same size, creating a slice index for each slice according to the row and column where the slice is located, and saving all slice textures obtained by cutting. The purpose is to decompose the large-scale panoramic image into smaller, manageable slices, and complete the mapping relationship between the longitude and latitude of the slice on the sphere and the coordinates in the panorama, so as to load and display it efficiently.

[0109] Preferably, the slice is a quadrilateral. In some embodiments, the slice is a square or a rectangle, and the length or width is a power of 2. In some embodiments, the slice is a square of 256 pixels×256 pixels or 512 pixels×512 pixels, and the side length of the square slice is denoted by T L .

[0110] In this embodiment, the slice is a square with a side length of 256 pixels, which can achieve better loading efficiency.

[0111] Furthermore, the total number of levels L of the panorama image to be cut is determined according to the slice size, and the slice size is the size of the slice in the Lth layer. The total number of levels of cutting can be 0, 1, 2, ... L, ... ...; when the total number of levels of cutting is L, the panorama is cut into 2 L ×2 Lslices, such as 1×1 slices when L=0 and 2×2 slices when L=1. Thus, the size of the panorama divided by the slice size gives the number of slices 2 L ×2 L , and further calculate the total number of levels L.

[0112] Preferably, L is 7 or 8, at which point the slice can provide sufficient image details without causing excessive loading and storage burdens. If the image needs to be displayed on a mobile device or the network bandwidth is limited, consider reducing the number of layers to improve loading efficiency.

[0113] Furthermore, the corresponding relationship between the slice index and its position coordinates in the panorama is determined. For the xth row and yth column, the slice index is (x tile ,y tile ), x tile and tile The value range is 0 to 2 L -1.

[0114] The panorama coordinates are the same as the pixel coordinates on the screen. The range starts from (0, 0) and the slice (x tile ,y tile ) is calculated as follows:

[0115] If the slice is rectangular, slice(x tile ,y tile ) ranges from (x tile ×tile_height,y tile ×tile_width) to ((x tile +1)×tile_height, (y tile +1)×tile_width), where tile_width is the width of the slice and tile_height is the height of the slice.

[0116] If the slice is square, the slice pixels range from (x tile ×T L ,y tile ×T L ) to ((x tile +1)×T L , (y tile +1)×T L ), where T L is the side length of the slice, such as 256 pixels. The coordinates of the center point of the slice are defined as (u, v).

[0117] Furthermore, the sphere is divided into multiple grids, and the longitude and latitude are used to represent the position on the sphere. The number and size of the grids on the sphere are consistent with the number and size of the slices divided by the background panorama. There is a one-to-one conversion relationship between the panorama coordinates and the longitude and latitude of the sphere. The longitude and latitude of the center point of the slice on the sphere can be calculated by the coordinates of the center point of the slice (u, v), and the calculation method is:

[0118]

[0119] Among them, (u, v) are the position coordinates of the center point of the slice in the panorama; u is the two-dimensional horizontal coordinate of the slice in the background panorama, and v is the two-dimensional vertical coordinate of the slice in the background panorama; lon is the longitude of the slice in the sphere; lat is the latitude of the slice in the sphere; w is the width of the background panorama; h is the height of the background panorama.

[0120] Conversely, when restoring the background panorama to the sphere, the corresponding background panorama slice is reversely searched from the spherical grid longitude and latitude information. The method of converting longitude and latitude to panorama coordinates is:

[0121]

[0122] Among them, (x tile ,y tile ) is the slice index in the background panorama, lon is the longitude of the slice in the sphere; lat is the latitude of the slice in the sphere, and L is the total number of cut levels of the background panorama.

[0123] Through the above coordinate transformation relationship, the matching relationship between the sphere grid and the background panorama slice is established. For the subsequent object tracking operation after the panorama is restored, the content of the grid can be easily adjusted according to the instructions.

[0124] In this embodiment, S400 specifically includes the following sub-steps:

[0125] Sub-step S410, cutting the background panorama: cutting the background panorama into square slices of the same size, the slice size is 256 pixels * 256 pixels, the slice index of the xth row and yth column is (x tile ,y tile ).

[0126] Sub-step S420: creating a new 3D scene object for placing all 3D objects.

[0127] Sub-step S430, create a camera and set the camera position. Preferably, in this embodiment, create a perspective camera with a field of view of 75 degrees, an aspect ratio of the window size, a near clipping plane of 0.1, and a far clipping plane of 1000. Set the camera position: Set the camera position at the origin.

[0128] Sub-step S440, create a renderer: set the renderer size and attach the renderer to the document. In this embodiment, a WebGL renderer is created, the size of the renderer is set to the width and height of the browser window, and the DOM element of the renderer is added to the document to make it visible for rendering the scene in the browser.

[0129] Sub-step S450: creating a sphere, and applying the background panoramic image to the sphere.

[0130] The specific process is as follows:

[0131] (1) Create a spherical geometry. Preferably, create a spherical geometry with a radius of 500, using 60 longitudinal segments and 40 transverse segments to increase the details of the sphere.

[0132] (2) Loading texture atlas: Create a texture loader and use it to load the texture atlas of the specified path. This atlas contains multiple small textures, which are used to replace different slices on the sphere;

[0133] (3) Create a material: Create a basic mesh material and use the loaded texture atlas to render the inner surface of the sphere. The main reason for rendering the inner surface is that the observer is looking at the sphere surface from inside the sphere, which is mainly used to create a 360-degree panoramic effect. By rendering from the inside perspective of the sphere, you can see all directions of the inner surface of the sphere;

[0134] (4) Create sphere mesh: Combine the sphere geometry and material into a mesh object to form a sphere. The number and size of the sphere mesh are consistent with the number and size of the slices divided into the background panorama.

[0135] (5) Add the sphere to the scene: Add the created sphere to the Three.js scene for rendering.

[0136] (6) Define a function to replace a specified slice: Define a function replaceSlice to replace the texture of a specified slice with the texture of the spherical mesh. This function accepts two parameters: the slice index and the new texture path. The new texture path stores the slice texture corresponding to the slice index.

[0137] When the longitude and latitude of the sphere grid are specified, the longitude and latitude are converted into slice indices, and the path of the corresponding slice texture is obtained through the slice index. By replacing the function of the specified slice, the texture at the corresponding position on the sphere is replaced with the corresponding slice, which is used to realize dynamic panoramic scene interactive applications.

[0138] The spherical mesh is traversed and looped, and the mesh textures are replaced one by one with the corresponding slices of the background panorama until all meshes and slices are replaced to restore the background.

[0139] In this embodiment, the panoramic image is loaded through threerjs, and THREE.TextureLoader is used to load the panoramic image.

[0140] S500: Merge the object to be tracked into the background panorama, and update the texture of the grid in the corresponding area of ​​the sphere. The method is: calculate the index of a group of slices within the coverage range of the contour of the object to be tracked and the displacement of the object to be tracked in the slice, realize the mapping relationship between the three-dimensional coordinates and viewing angle information of the object to be tracked and the specific position on the two-dimensional plane of the background panorama, that is, determine the slice position of the object in the background panorama.

[0141] The specific steps include:

[0142] S510 calculates the two-dimensional view coordinates of the object to be tracked under the camera's view angle. This step converts the three-dimensional coordinates of the object into two-dimensional coordinates under the camera's view angle through the camera's projection matrix and viewport matrix. The specific method is:

[0143] First, convert the three-dimensional world coordinates (Xw, Yw, Zw) of the center point of the object to be tracked obtained in the step into the coordinates of the camera coordinate system using the following formula:

[0144] Camera_Coord=View_Matrix×World_Coord

[0145] World_Coord is the coordinate vector (Xw, Yw, Zw) of the object in the world coordinate system, View_Matrix is ​​the view matrix of the camera, and Camera_Coord is the coordinate of the object to be tracked in the camera coordinate system.

[0146] Then, the coordinates of the object to be tracked in the camera coordinate system are converted into two-dimensional view coordinates under the camera's perspective through the projection matrix. The conversion formula is as follows:

[0147] Clip_Coord=Projection_Matrix×Camera_Coord

[0148] Projection_Matrix is ​​the projection matrix of the camera, and Clip_Coord is the two-dimensional perspective coordinates of the object from the camera's perspective.

[0149] S520, converting the two-dimensional view coordinates Clip_Coord of the object under the camera view into the pixel coordinates of the object on the screen, the calculation method is:

[0150]

[0151] Among them, (Clip_Coord x, Clip_Coord y ) is the two-dimensional viewing coordinate of the object under the camera’s viewing angle; (x norm ,y norm ) is the normalized screen coordinate, ranging from -1 to 1; Clip_Coord w are the homogeneous coordinate components used for perspective division;

[0152] (x screen ,y screen ) is the pixel coordinate of the object on the screen; Screen_Width is the width of the screen, Screen_Height is the width of the screen, and the units are all pixels.

[0153] S530, mapping the pixel coordinates of the object on the screen to the slice. The position of the object on the screen is obtained according to the coordinates of the center point of the object, the angle between the center point of the object and camera A, and the scale of the model, and the slice position of the object in the background panorama is calculated. The slice position of the object includes two parameters: the slice index and the offset of the object in the slice.

[0154] The specific method is: Assume that the spherical background image is divided into several slices, and the width and height of each slice are tile_width and tile_height respectively. Then the screen coordinates of the object can be mapped to the corresponding slice index of the object in the background panorama.

[0155]

[0156] Among them, (x screen ,y screen ) is the pixel coordinate of the object on the screen; (x tile ,y tile ) is the slice index in the background panorama, tile_width is the slice width, tile_height is the slice height, and floor is a rounding operation. In this embodiment, tile_width and tile_height are the same, which is 256 pixels.

[0157] Furthermore, the offset of the object in the slice is calculated to determine the specific position of the object relative to the slice:

[0158] x offset =x screen modtile_width

[0159] y offset =y screen modtile_height

[0160] tile_width is the width of the tile; tile_height is the height of the tile; x offset is the horizontal offset of the object in the slice; offset It is the longitudinal offset of the object in the slice; mod is the remainder operation.

[0161] S540, according to the slice position of the object to be tracked in the background image, the panoramic image of the object to be tracked is loaded onto the background panoramic image, the slice image of the area where the object is located is regenerated, and the grid texture of the area where the object is located on the sphere is replaced.

[0162] We need to note that since the background panorama is an absolute coordinate starting from (0, 0), and the slice coordinates correspond to the longitude and latitude information of the sphere grid, as long as the longitude and latitude corresponding to the center point of each slice are determined, the background can be restored slice by slice. The coordinates of the object are determined by its relative position in the background, so the coordinate conversion method when restoring the object to be tracked is different from that of the background.

[0163] Step S600: Setting tracking operations for the objects to be tracked. Specifically, adding interactive events for all objects to be tracked in the scene, and setting all objects to be deleted, replaced, and moved. Related events are triggered by screen input operations obtained by the system, and deletion, replacement, or movement operations are performed based on the correspondence between the background, the objects to be tracked, the replacement objects, and the spherical grid.

[0164] The present invention divides the background panorama into multiple slices. The objects to be tracked and the replacement objects only cover a certain slice area in the background. Deletion and replacement only involve the slice area occupied by the objects to be tracked and the replacement objects, thereby reducing overhead and calculation amount and improving efficiency.

[0165] The method of deleting the object to be tracked is as follows: when the system obtains the operation instruction to delete the object to be tracked, a pop-up box will pop up to confirm the deletion (the pop-up box is to avoid misoperation). If the deletion is confirmed, all the slice indexes in the area covered by the object to be tracked in the panorama are calculated, and the corresponding sphere mesh texture is updated to the corresponding slice of the background panorama according to the slice index. In the initial 3D home improvement design plan, the specific operation corresponding to the deletion is: obtaining the basic information and coordinate information of the object to be tracked, hiding the object to be tracked and the corresponding replacement object to be deleted, and displaying the background and other object information.

[0166] Replacement operation: After the system obtains the operation of replacing the object to be tracked, it displays a group of replacement objects corresponding to the object to be tracked, obtains the information of a selected replacement object, calculates the slice area covered by the object to be tracked and the replacement object, first performs the operation of deleting the object to be tracked, and then restores the replacement object to the background. Here, the method of step S500 can be used to restore the replacement object to the background.

[0167] Operation of moving the object to be tracked: obtain the operation information of dragging the object to be tracked on the screen. When the target position of the object to be tracked exceeds the movable range, it will remind you that the moving operation cannot be completed; when the target position of the object to be tracked falls within the movable range, the operation of deleting the object to be tracked is executed first, and then the object to be tracked is restored to the background according to the target position of the object to be tracked on the screen.

[0168] The method to determine the movable range of the object to be tracked on the screen is:

[0169] (1) First, the movable range of the object to be tracked in the three-dimensional scene of the home decoration design plan is obtained, and the three-dimensional coordinates of the bounding box of the moving range of the object to be tracked are obtained.

[0170] The movable range of the object to be tracked is usually calculated based on factors such as the position of the object to be tracked in three-dimensional space, the size of the bounding box, the current viewing angle, and the camera field of view.

[0171] Factors that affect the draggable range are:

[0172] Object properties: The size, type, and category of an object affect the area in which it can be dragged. For example, in some embodiments, large furniture is not allowed to cross certain obstacles or physical limitations.

[0173] Environmental factors: The environment in which the object is located, such as the walls, ground or other elements in the scene, will also affect the movement range of the object. It may be necessary to calculate the draggable area based on the scene elements. Camera perspective and projection: The current perspective and projection matrix affect the position and visible range of the object on the screen. In some embodiments, when the camera's perspective is used to limit the movement range of the object, the object is restricted to always remain within the visible area.

[0174] In this embodiment, in order to simplify the scheme, the movable range of each object to be tracked is preset as a rectangular bounding box and the three-dimensional coordinates of each corner point of the bounding box are given, that is, the three-dimensional coordinates of the bounding box of the moving range of the object to be tracked. Among them, the minimum point is a corner point of the bounding box, which has the smallest x, y and z coordinate values. In Three.js, this point can be obtained by boundingBox.min and expressed as (x_min, y_min, z_min); Maximum point (max): This is another corner point of the bounding box diagonally opposite the minimum point, with the largest x, y and z coordinate values. In Three.js, this point can be obtained by boundingBox.max and expressed as (x_max, y_max, z_max).

[0175] (2) Convert the three-dimensional coordinates of the bounding box of the moving range of the object to be tracked into a movable area on the screen. The method is: use the camera's projection matrix to convert the three-dimensional coordinates of the bounding box into two-dimensional coordinates under the camera's perspective, and then further convert them into screen coordinates to obtain a rectangular area where the object can be dragged on the screen. The minimum point of the rectangle is defined as (min_x, min_y), and the maximum point is defined as (max_x, max_y). The coordinate conversion method is recorded in step S500.

[0176] When dragging the object to be tracked, first determine whether the object to be tracked is within the rectangular area between the draggable range (min_x, min_y) and (max_x, max_y). If it is within the range, it can be dragged. If it is not within the range, it will be prompted that it cannot be dragged. If it is within the range, when you stop dragging, first delete the original object to be tracked, refer to the S600 deletion operation step method; then determine the target position of the object to be tracked based on the screen coordinates when the dragging stops, refer to the method of S500, restore the object to be tracked to the target position, regenerate the panoramic slice in the new area covered by the object to be tracked, and update the texture of the corresponding set of grids on the sphere.

[0177] Step S700: Loading a sphere obtained by merging the background panorama and the panorama of all objects to be tracked into a browser.

[0178] Step S800: Obtain the location information of the user's mouse click through the browser; determine whether the mouse click element is the object to be tracked, if not, skip it; if it is the object to be tracked, the operation options of the object to be tracked will appear: delete, replace and move, and apply the method recorded in S600 to perform the corresponding operation, and provide real-time feedback on the effect of the adjustment of the object to be tracked through slicing update; repeat the above operations until the solution modification is completed and the object tracking is completed.

[0179] After the modification of the plan is completed, the customer confirms and saves the final design plan. Furthermore, the panoramic image position corresponding to the object in the final design plan is restored to the three-dimensional coordinates through coordinate conversion, and the home decoration design plan is updated.

[0180] like Figures 2 to 4 As shown, this embodiment provides a complete tracking process. First, design a decoration plan in 3D design software (any design software such as 3Dmax or Maya), and classify the decoration elements in the decoration plan. Among them, the decoration elements that can be deleted, replaced or moved are classified as objects to be tracked, and the remaining decoration elements are classified as scenes as a whole, and one or more replacement objects are set for the objects to be tracked. The above method forms a home decoration design plan to be processed, and the home decoration design plan includes two types of elements: background and object. Then, the background and objects are rendered separately to generate a background panorama and a panorama of the object. The design plan is restored according to the background panorama and the panorama of the object. As shown in Figure 5 As shown, when the object to be tracked in the panoramic image needs to be replaced, only the image of the area covered by the object to be tracked is sliced ​​and replaced.

[0181] like Figure 6 and Figure 7 As shown, a plurality of replaceable coffee tables are provided for selection, and the user can click on the coffee table to delete, replace or move it according to the needs.

[0182] The present invention cuts the panorama into small, manageable slices, completes the mapping between the slice positions in the panorama and the longitude and latitude in the sphere, and in the interactive feedback process of scheme update, efficient loading and display of images in the object tracking process is achieved by updating some slices in the panorama, so that the panorama of the home improvement design scheme can be edited and adjusted in real time with visual adjustment, thereby improving the user experience.

[0183] The present invention determines the conversion relationship between the three-dimensional coordinates of the object and the coordinates in the panoramic image through the relative position relationship between the object and the background, and obtains the position of the object in the slice. In the process of restoring the object to the background, it only needs to replace the slice of the area covered by the object with the corresponding part of the panoramic image after the background and the object are combined, thereby reducing the amount of calculation. Through the same method, the objects can be restored to the background panoramic image one by one, and the objects can be quickly deleted, moved and replaced, thereby realizing the boundary editing of the home decoration design plan.

[0184] In this way, through the form of slicing and segmentation, during the object tracking process, it is possible to track multiple objects in a large space without the need to rebuild the panorama on a large scale, and to delete, replace and move the tracked objects in the panorama in real time and conveniently, making the editing of home decoration design plans simple and efficient.

[0185] The above description and the accompanying drawings fully illustrate the embodiments of the present disclosure so that those skilled in the art can practice them. Other embodiments may include structural, logical, electrical, process and other changes. The embodiments represent only possible changes. Unless explicitly required, separate components and functions are optional, and the order of operation may vary. The parts and features of some embodiments may be included in or replace the parts and features of other embodiments. Moreover, the words used in this application are only used to describe the embodiments and are not used to limit the claims. As used in the description of the embodiments and the claims, unless the context clearly indicates, the singular forms of "a", "an" and "the" are intended to include plural forms as well. Similarly, the term "and / or" as used in this application refers to any and all possible combinations of listings containing one or more associated ones. In addition, when used in the present application, the term "comprise" and its variants "comprises" and / or comprising refer to the presence of stated features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method or device including the elements. In this article, each embodiment may focus on the differences from other embodiments, and the same and similar parts between the embodiments may refer to each other. For the method disclosed in the embodiment, if it corresponds to the method part disclosed in the embodiment, then the relevant parts can refer to the description of the method part.

[0186] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software may depend on the specific application and design constraints of the technical solution. The technicians may use different methods for each specific application to implement the described functions, but such implementations should not be considered to exceed the scope of the embodiments of the present disclosure. The technicians may clearly understand that, for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above may refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

Claims

1. A method for object tracking based on a 360-degree panoramic image, the steps are as follows: Step S100: design a 3D home decoration design plan, which includes decoration elements, select some or all of the decoration elements as objects to be tracked, and mark the remaining elements as a background; The object to be tracked is an element in the design scheme that is preset to support deletion, replacement and movement operations; Step S200: setting at least one replacement object for each object to be tracked in step S100, wherein the replacement object and the object to be tracked are stored in different layers of the design scheme, and the center point of the replacement object coincides with the center point of the object to be tracked; Replacement objects are decorative elements used to replace original objects in the design plan; Step S300: setting the camera position, rendering the background, the object to be tracked and the replacement object in the home decoration design scheme based on the same camera position, and generating a background panorama, a panorama of the object to be tracked and a panorama of the replacement object; Record the coordinates of the center point of the replacement object (Xw, Yw, Zw), the angle between the center point of the replacement object and the camera, and the scale of the model; Step S400: Load the background panorama into the 3D sphere to complete the restoration of the background of the home decoration design plan; the method is: create a 3D scene through 3D design software to place all 3D objects; create a camera and set the camera position; create a renderer to render the scene in the browser; Create a sphere, load the background panorama into the sphere, and restore the background of the home decoration design plan; Before loading the background panorama into the sphere, cut the background panorama into slices of the same size, and establish a slice index (xtile, ytile) for the slice located at the x row and y column; save all the slice textures obtained by cutting; divide the sphere into multiple grids, and use longitude and latitude to represent the position on the sphere; the number and size of the sphere grids are consistent with the number and size of the slices divided by the background panorama; when specifying the longitude and latitude of the sphere grid, first convert the longitude and latitude into a slice index, and obtain the path of the corresponding slice texture through the slice index, and replace the texture of the corresponding position on the sphere with the corresponding slice texture by replacing the function of the specified slice. When replacing, convert the coordinates of the center point of the slice into longitude and latitude to accurately locate the slice to the sphere grid; traverse the sphere grid and replace the grid textures one by one with the texture of the corresponding slice in the background panorama until all grids are replaced to restore the background; Step S500: all objects to be tracked are merged into the background panorama, and the texture of the grid corresponding to the coverage area of ​​the object to be tracked on the sphere is updated; the method is: first, according to the mapping relationship between the three-dimensional coordinates and the viewing angle information of the object to be tracked and the specific position on the two-dimensional plane of the background panorama, the index of a group of slices within the coverage range of the contour of the object to be tracked and the displacement of the object to be tracked in the slice are calculated, and the slice position of the object to be tracked in the background panorama is obtained, and then according to the slice position, the panorama of the object to be tracked is merged into the background panorama; Finally, the merged panorama is sliced ​​according to the same rules as the background panorama to obtain a set of new slices within the coverage of the object to be tracked, which are used to update the texture of the grid in the corresponding area of ​​the sphere; Repeat the above steps until all objects to be tracked are restored; Step S600: Setting tracking operations for the objects to be tracked; specifically, adding interactive events for all the objects to be tracked in the scene, the interactive events including deletion, replacement and movement operations; the relevant events are triggered by the screen input operation acquired by the system, and the deletion, replacement or movement operations of the objects to be tracked are performed according to the relationship between the background panorama, the panorama of the object to be tracked and the panorama of the replacement object, and the correspondence between the panorama slices and the spherical grid; Step S700: loading a sphere obtained by merging the background panorama and the panorama of all objects to be tracked into a browser; Step S800: obtaining the location information of the user's mouse click through the browser; Determine whether the mouse clicked element is the object to be tracked, if not, skip it; If it is an object to be tracked, operation options for the object to be tracked appear: delete, replace and move, and the method recorded in S600 is applied to perform corresponding operations, and the effect of the adjusted object to be tracked is fed back in real time through slice update; the above operations are repeated until the solution modification is completed and the object tracking is completed.

2. The object tracking method based on a 360-degree panoramic image according to claim 1, characterized in that: Step S400 also includes the following steps: determining the total number of slice levels L of the panoramic image according to the slice size, and the calculation method is: the size of the panoramic image is divided by the slice size to obtain the number of slices 2 L ×2 L ;x tile and tile The value range is 0 to 2 L -1; The coordinates of the slice center point in the background panorama are defined as (u, v).

3. The object tracking method based on a 360-degree panoramic image according to claim 2, characterized in that: In step S400, there is a one-to-one conversion relationship between the background panorama coordinates and the longitude and latitude of the sphere. The longitude and latitude of the center point of the slice on the sphere can be calculated by the coordinates (u, v) of the center point of the slice. The calculation method is: Wherein, (u, v) is the position coordinate of the center point of the slice in the background panorama; u is the two-dimensional horizontal coordinate of the slice in the background panorama, and v is the two-dimensional vertical coordinate of the slice in the background panorama; lon is the longitude of the slice in the sphere; lat is the latitude of the slice in the sphere; w is the width of the background panorama; h is the height of the background panorama; The method of converting spherical longitude and latitude into background panorama slice index is: Among them, (x tile ,y tile ) is the slice index in the background panorama, lon is the longitude of the slice in the sphere; lat is the latitude of the slice in the sphere, and L is the total number of cut levels of the background panorama.

4. The object tracking method based on a 360-degree panoramic image according to claim 3, characterized in that: In step S400, the specific process of applying the background panoramic image to the sphere is as follows: (1) Create a sphere geometry; (2) Loading a texture atlas: Create a texture loader and use it to load a texture atlas of a specified path. This atlas contains multiple small textures for replacing different slices on the sphere. (3) Create a material: Create a basic mesh material and use the loaded texture atlas to render the inner surface of the sphere; (4) Creating a sphere mesh: The sphere geometry and material are combined into a mesh object to form a sphere; the sphere is divided into grids with the same number and size as the background panorama slices; (5) Add the sphere to the scene: Add the created sphere to the 3D scene; (6) Define a function to replace a specified slice: Define a function replaceSlice to replace the texture of a specified slice with the texture of the spherical mesh. This function accepts two parameters: the slice index and the new texture path. The new texture path stores the slice texture corresponding to the slice index.

5. The object tracking method based on a 360-degree panoramic image according to claim 4, characterized in that: Step S500 includes the following sub-steps: In sub-step S510, the three-dimensional world coordinates (Xw, Yw, Zw) of the center point of the object to be tracked are converted into coordinates in the camera coordinate system by the following formula: Camera_Coord=View_Matrix×World_Coord World_Coord is the coordinate vector (Xw, Yw, Zw) of the object in the world coordinate system, View_Matrix is ​​the view matrix of the camera, and Camera_Coord is the coordinate of the object to be tracked in the camera coordinate system; Then, the coordinates of the object to be tracked in the camera coordinate system are converted into two-dimensional view coordinates under the camera's perspective through the projection matrix. The conversion formula is as follows: Clip_Coord=Projection_Matrix×Camera_Coord Projection_Matrix is ​​the projection matrix of the camera, Clip_Coord is the two-dimensional perspective coordinates of the object to be tracked under the camera's perspective; Sub-step S520, converting the two-dimensional view coordinates Clip_Coord of the object to be tracked under the camera view into the pixel coordinates of the object to be tracked on the screen, the calculation method is: Among them, (Clip_Coord x , Clip_Coord y ) is the two-dimensional viewing coordinate of the object under the camera’s viewing angle; (x norm ,y norm ) is the normalized screen coordinate, ranging from -1 to 1; Clip_Coord w are the homogeneous coordinate components used for perspective division; (x screen ,y screen ) is the pixel coordinate of the object on the screen; Screen_Width is the width of the screen, Screen_Height is the height of the screen, both in pixels; Sub-step S530, mapping the pixel coordinates of the object to be tracked on the screen to the slice; obtaining the position of the object to be tracked on the screen according to the coordinates of the center point of the object to be tracked, the angle between the center point of the object and camera A, and the scaling ratio of the model, and calculating the slice position of the object in the background panorama, the slice position of the object includes two parameters: the slice index and the offset of the object in the slice; The specific method is: Among them, (x screen ,y screen ) is the pixel coordinate of the object on the screen; (x tile ,y tile ) is the slice index in the background panorama, tile_width is the slice width, tile_height is the slice height, and floor is the rounding operation; x offset =x screen mod tile_width y offset =y screen mod tile_height tile_width is the width of the tile; tile_height is the height of the tile; x offset is the horizontal offset of the object in the slice; offset is the longitudinal offset of the object in the slice; mod is the remainder operation; Sub-step S540, according to the slice position of the object to be tracked in the background image, the panoramic image of the object to be tracked is loaded onto the background panoramic image, the slice image of the area where the object is located is regenerated, and the grid texture of the area covered by the object to be tracked on the sphere is replaced.

6. The object tracking method based on a 360-degree panoramic image according to claim 5, characterized in that: In step S600, the method for deleting the object to be tracked is: calculating all slice indexes in the coverage area of ​​the object to be tracked in the panoramic image, and updating the corresponding spherical mesh texture to the corresponding slice of the background panoramic image according to the slice index; Replacement operation: After the system obtains the operation of replacing the object to be tracked, it displays a group of replacement objects corresponding to the object to be tracked, obtains the information of a selected replacement object, calculates the slice area covered by the object to be tracked and the replacement object, first executes the operation of deleting the object to be tracked, and applies the method of step S500 to restore the replacement object to the background.

7. The object tracking method based on a 360-degree panoramic image according to claim 6, characterized in that: In step S600, the operation method for moving the object to be tracked is: obtaining information on dragging the object to be tracked on the screen, and when the target position of the object to be tracked exceeds the movable range, a reminder is given that the moving operation cannot be completed; when the target position of the object to be tracked falls within the movable range, the operation of deleting the object to be tracked is first performed, and then the method of step S500 is applied to restore the object to be tracked to the target position of the background panorama, regenerate the panoramic slice in the new area covered by the object to be tracked, and update the texture of the corresponding set of grids on the sphere.

8. The object tracking method based on a 360-degree panoramic image according to claim 7, characterized in that: In step S600, the method for determining the movable range of the object to be tracked on the screen is: first, obtain the preset movable range of the object to be tracked in the three-dimensional scene of the home decoration design plan, and obtain the three-dimensional coordinates of the bounding box of the moving range of the object to be tracked; convert the three-dimensional coordinates of the bounding box of the moving range of the object to be tracked into a movable area on the screen; the method is: use the camera's projection matrix to convert the three-dimensional coordinates of the bounding box into two-dimensional coordinates under the camera's perspective, and then further convert them into screen coordinates to obtain a rectangular area on the screen where the object to be tracked can be dragged, and define the minimum point of the rectangle as (min_x, min_y), and the maximum point as (max_x, max_y); the method for converting from three-dimensional coordinates to pixel coordinates on the screen is recorded in step S500.

9. The object tracking method based on a 360-degree panoramic image according to claim 8, characterized in that: After step S800, the following steps are also included: after the plan modification is completed, the customer confirms and saves the final design plan; the panoramic image position corresponding to the object in the final design plan is restored to the three-dimensional coordinate through coordinate conversion, and the home improvement design plan is updated.

10. The object tracking method based on a 360-degree panoramic image according to claim 9, characterized in that: In step S100, the 3D design software is 3dsMax software, and in step S400, THREE.TextureLoader of threerjs software is used to load the panoramic image.

Images