Parallel method and device for flood filling

Abstract

A flood-fill method for parallel implementation comprises the steps of: receiving an image, the image comprising a first element having a flood-fill source color to be replaced with a flood-fill target color, and a second element not having the flood-fill source color; partitioning the image according to a hierarchical space partitioning scheme to form cells at a plurality of levels in the hierarchy; detecting the occupancy of each cell by a second element; for any cell not occupied by the second element, setting the depth-elevation value of a child cell of that cell to a depth-elevation value greater than the value of that cell, indicating that the flood-fill should navigate the hierarchy toward a root cell by the cell's depth-elevation amount while occupying only nodes without any second element, thereby reaching the largest parent cell that can be flood-filled; and filling the flood-fill source color of cells not occupied by the second element with the flood-fill target color, thereby modifying the received image.

Description

Technical Field

[0001] The invention relates to a parallel method of flood filling and a corresponding device. Background Art

[0002] 3D printing is a means of three-dimensional printing, in which the printer deposits three-dimensional voxels in a volume instead of two-dimensional pixels on a flat surface. There are a variety of techniques to achieve this, including using lasers to melt and fuse materials such as metal powders layer by layer to build structures, or using high-temperature print heads to deposit small droplets of plastic that cool and harden to build 3D models.

[0003] The design of such 3D models is usually generated in a computer-aided design program, in which the user defines the surfaces and volumes of the virtual model. The program then generates a draw list, which specifies the order in which the material is to be melted or deposited using a 3D printer to render the virtual model.

[0004] This approach has led to the creation of many aesthetically and functionally unique objects, some of which are difficult to manufacture using traditional processes. However, the use of 3D printers remains underexplored. Summary of the invention

[0005] The present invention aims to provide a new use for a 3D printer.

[0006] In a first aspect, according to claim 1, there is provided a flood-fill method for parallel implementation.

[0007] In another aspect, according to claim 16, an entertainment device is provided.

[0008] Further aspects and features of the invention are defined in the appended claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Embodiments of the present invention will now be described by way of example with reference to the accompanying drawings, in which:

[0010] Figure 1 is a schematic diagram of an entertainment device according to an embodiment of the present invention.

[0011] Figure 2A is a schematic diagram of a side view of a 3D printer according to an embodiment of the present invention.

[0012] Figure 2B is a schematic diagram of a plan view of a 3D printer according to an embodiment of the present invention.

[0013] Figure 3 is a schematic diagram of a video game virtual environment according to an embodiment of the present invention.

[0014] Figure 4 is a schematic diagram of projection to a 3D model space according to an embodiment of the present invention.

[0015] Figure 5 Schematic diagram of a default virtual camera position for virtual photogrammetry processing according to an embodiment of the present invention.

[0016] Fig. 6A and 6B is a schematic diagram of a 3D model generated by virtual photogrammetry processing, rendered based on a default virtual camera position.

[0017] Figure 7 is a flowchart of a 3D printing modeling method according to an embodiment of the present invention.

[0018] Fig. 8A is an image of a target object according to an embodiment of the present invention.

[0019] Figure 8B is an image of a voxel model of the surface of a target object according to an embodiment of the present invention.

[0020] Fig. 9 is a flow chart of a 3D printing modeling method according to an embodiment of the present invention.

[0021] FIG. 10A to FIG. 10E is a schematic diagram illustrating a process of generating a geometric figure for 3D printing according to an embodiment of the present invention.

[0022] Fig.11 is a schematic diagram of a 3D printable video game character according to an embodiment of the present invention.

[0023] Fig.12 is a schematic diagram of a 3D printable video game character according to an embodiment of the present invention.

[0024] Fig.13A , Fig. 13B and Fig. 13C is a schematic diagram of a traditional flood-fill operation.

[0025] Fig.14 is a schematic diagram of a 2D image to be filled by Fan Hong according to an embodiment of the present invention.

[0026] Fig.15 It is a representation of the elevation depth value based on image occupancy within a cell of the hierarchical spatial segmentation scheme according to an embodiment of the present invention.

[0027] Fig.16 4 is a schematic diagram of elevation depth values ​​based on image occupancy within a unit of a hierarchical space segmentation scheme according to an embodiment of the present invention.

[0028] Fig.17AThe recursion of sub-units in a hierarchical space partitioning scheme according to an embodiment of the present invention is illustrated.

[0029] Fig. 17B The traversal of neighboring cells in a hierarchical space partitioning scheme according to an embodiment of the present invention is illustrated.

[0030] Fig.18A , Fig.18B and Fig. 18C is an illustration of an improper flood fill operation.

[0031] Fig.19A is an illustration of an object to be assembled for 3D printing.

[0032] Fig.19B , Fig.19C and Fig.19D is an illustration of an improper flood fill operation.

[0033] Fig. 20 is a diagram of a correct flooding operation according to an embodiment of the present invention.

[0034] Fig.21 a to Fig.21 h is a diagram illustrating modification of an image according to an embodiment of the present invention.

[0035] Fig. 22 is a flow chart of a method for parallel implementation of flood filling according to an embodiment of the present invention. DETAILED DESCRIPTION

[0036] A system and method for flood filling for parallel implementation is disclosed. In the following description, many specific details are provided to provide a thorough understanding of the embodiments of the present invention. However, it is apparent to those skilled in the art that these specific details are not required to practice the present invention. On the contrary, for the sake of clarity, specific details known to those skilled in the art have been omitted where appropriate.

[0037] In an embodiment of the present invention, the entertainment device is operatively coupled to a 3D printer. As described below, the entertainment device is arranged in operation to capture a snapshot of a video game for reproduction by the 3D printer.

[0038] Entertainment Equipment

[0039] Examples of suitable entertainment equipment are equipment.

[0040] Figure 1 Schematically illustrates Overall system architecture of an entertainment device. A system unit 10 is provided having various peripheral devices connectable to the system unit.

[0041] The system unit 10 includes an accelerated processing unit (APU) 20 as a single chip, which includes a central processing unit (CPU) 20A and a graphics processing unit (GPU) 20B. The GPU can implement multiple work products in parallel. The APU 20 can access a random access memory (RAM) unit 22.

[0042] APU 20 optionally communicates with bus 40 via I / O bridge 24 , which may be a discrete component of APU 20 or part of APU 20 .

[0043] Connected to bus 40 are data storage components such as hard disk drive 37 and Blu-ray drive 36 operable to access data on compatible optical disk 36A. Additionally, RAM unit 22 may be in communication with bus 40.

[0044] Also optionally connected to bus 40 is an auxiliary processor 38. Auxiliary processor 38 may be provided to run or support an operating system.

[0045] The system unit 10 communicates with peripheral devices as appropriate via an audio / video input port 31, an Ethernet port 32, a Bluetooth wireless link 33, a Wi-Fi wireless link 34, or one or more universal serial buses (USB). Audio and video may be output via an AV output 39 such as an HDMI port.

[0046] Peripheral devices may include a monoscopic or stereoscopic camera 41, such as a PlayStation Eye or PS Camera; a stick-style video game controller 42, such as a PlayStation Move, and a traditional handheld video game controller 43, such as a Portable entertainment devices 44, such as and A keyboard 45 and / or a mouse 46; a media controller 47, for example in the form of a remote control; and a headset 48. Other peripherals such as a phone or tablet, a printer, or a 3D printer (not shown) may similarly be considered.

[0047] GPU 20B, optionally in conjunction with CPU 20A, generates video images and audio for output via AV output 39. Optionally, audio may be generated in conjunction with or by an audio processor (not shown).

[0048] The video and audio (optionally) may be presented to a television 51. The video may be stereoscopic where supported by the television. The audio may be presented to a home theater system 52 in one of a variety of formats such as stereo, 5.1 surround sound, or 7.1 surround sound. The video and audio may also be presented to a head mounted display unit 53 worn by a user 60. The head mounted display unit may have integrated headphones, attachable headphones / earbuds, or rely on separate audio provided to the user.

[0049] In operation, the entertainment device defaults to an operating system such as a variant of FreeBSD 9.0. The operating system may run on the CPU 20A, the auxiliary processor 38, or a mixture of the two. The operating system provides a graphical user interface to the user, such as the PlayStation dynamic menu. The menu allows the user to access operating system functions and select games and optionally other content.

[0050] 3D Printer

[0051] As previously mentioned, the entertainment device is operably coupled to the 3D printer.

[0052] It should be understood that there are many technologies for 3D printing, which are generally distinguished by how the layers are built to create the model. One technology is called selective laser sintering (SLS), in which a layer of powder is selectively fused together by a laser to form a solid area; a new layer of powder is then placed on top and the process is repeated to build the model. Another technology is called stereolithography, and uses a photoreactive liquid in conjunction with one or more lasers to harden the liquid at defined points, again typically building the model in layers. While both of these technologies are contemplated within the scope of embodiments of the present invention, their disadvantages are the need for high-powered lasers and large amounts of matrix material in the form of powder or liquid surrounding the model, which makes them less practical for home use. Therefore, the preferred technology is fused deposition modeling (FDM). This method melts the plastic in a print head that moves over successive layers of the model, depositing droplets of plastic at selected locations in each layer in a manner similar to the way ink droplets are deposited on paper by an inkjet printer. This avoids the need for a laser or a surrounding matrix of the raw material used for the model. Therefore, for ease of understanding, references herein to fused deposition modeling are made. Figure 2A and 2B A brief description of FDM 3D printers.

[0053] Figure 2A 1 shows a side view of a simple FDM 3D printer 100, and Figure 2BA plan view of the same FDM 3D printer is shown. The printer 100 comprises a base structure 110 which provides a working surface for assembly of printed models, and support posts 102 upon which a printer frame 120 can be raised and lowered.

[0054] In the example mechanism, the motor 112Y is coupled to the drive rod 104 including threads; then, the printer frame 120 includes a coupling portion having a threaded hole and a smooth hole, the threaded hole engages with the drive rod, and the smooth hole engages with one of the support posts. When the motor 112Y rotates the drive rod in a clockwise or counterclockwise direction under the instruction of the printer driver, the printer frame moves the drive rod and the support post up or down (i.e., along the Y axis), and thereby raises or lowers the printer carriage 130 attached thereto.

[0055] like Figure 2B As shown, the printer frame 120 is typically mounted on four support posts 102. Optionally, a second motor, drive rod and coupling portion may be provided at an opposite corner of the frame to reduce stress on the frame and motor.

[0056] The printer frame 120 supports a printer carriage 130. A motor 112X is coupled to a drive belt 122 secured to the printer carriage 130 by means 124. When the motor 112X rotates the drive belt clockwise or counterclockwise under the direction of the printer driver, the printer carriage 130 moves left or right along the printer frame 120 (i.e., along the X-axis), and thereby moves the attached printer mechanism 140 laterally.

[0057] The printer carriage 130 supports a printer mechanism 140. The motor 112Z is ​​coupled to a drive belt 132 secured to the printer mechanism 140 by means 134. When the motor 112Z rotates the drive belt clockwise or counterclockwise under the direction of the printer driver, the printer mechanism 140 moves inwardly or outwardly in the depth direction (i.e., along the Z axis).

[0058] The printer mechanism 140 itself includes heating means for heating the ends of one or more plastic threads fed into the mechanism (not shown), or for heating plastic powder pellets obtained from a hopper on the mechanism (not shown). The heating of the plastic, or alternatively the release of heated plastic, is controlled by instructions from the printer driver. The resulting beads or drops of molten plastic are then deposited by a print head or nozzle 142 onto the working surface 110 of the printer or onto a partially built model (not shown), as applicable.

[0059] Thus, under appropriate instructions from the printer driver, the print head can be placed at any position within the working volume of the 3D printer using motors 112X, Y, Z, and a drop of plastic can be deposited at that position, which then cools and hardens to form a voxel of the 3D model. By continuous movement of the print head and selective melting or release of the drop of plastic, a complete 3D model can thus be constructed from a plurality of such voxels.

[0060] Typically, the printer driver itself is a software module in a computer-aided design system that receives the model geometry describing the 3D model. The printer driver then generates a voxel-thick slice of the 3D model for each layer in the y direction and determines the x, z coordinates of each voxel in that layer. The printer driver then outputs a series of instructions to the printer 100 to move the printer head 142 to the respective x, z coordinates of each voxel in layer y, instructing the printer mechanism to heat and / or release a drop of plastic to form a voxel at that location. In this way, the 3D printer reconstructs the digital 3D model into a physical model.

[0061] In an embodiment of the present invention, a printer driver is incorporated into the operating system of a video game or entertainment device, or into a middleware library of supporting functions used by the video game or operating system.

[0062] Virtual Environment

[0063] Reference now Figure 3 , a video game run on an entertainment device includes a virtual environment in which multiple entities or environmental elements are typically viewed dynamically as a user changes the position of a viewpoint and as the game entities or elements perform scripted activities or react to the user's actions, or both.

[0064] exist Figure 3 In FIG. 2 , a simple environment 200 is shown, which includes a room 210 with a door 212; on one wall, candles are mounted in candlesticks 214. In the room is a player's character 220 (here shown, for example, by the fictional game character Blobman).

[0065] The virtual environment is constructed in 3D from geometric data, typically in the form of polygons that define the surfaces of objects. These polygons may be predefined, for example in the case of static objects and background scenes, or may be distorted / repositioned or procedurally generated in the case of mobile entities within the environment (e.g., a player's character). It should be understood that references to "polygons" herein include initial geometric features, such as vertices from which polygons are constructed, where these polygons are used in the graphics pipeline. Similarly, voxel rendering uses equivalent geometric features to describe objects. Therefore, processing described as applied to polygons may be applied in whole or in part to such other geometric features as appropriate.

[0066] The virtual environment is then prepared for presentation to the user from the selected viewpoint. Environmental elements to which the user has line of sight are patterned using texture information appropriate to the objects they represent, and the textured geometry will further handle lighting effects, such as brightness variations with distance to virtual light sources, and bump mapping (or similar techniques such as displacement mapping or using isosurfaces) to simulate how the texture should interact with such virtual light sources. In addition, other effects such as volumetric fog and particle effects may also be included.

[0067] The final rendered image is then typically presented to the user via a 2D or 3D television or via a head-mounted display.

[0068] Often in such games, users have experiences that they wish to share with friends and / or the wider world. To this end, as previously mentioned, entertainment devices such as the PlayStation 4 may have a "share" button on its controller to facilitate capturing images presented to the user, which may then be uploaded to a sharing site, such as a social media site.

[0069] Print 3D models of environments from image collections

[0070] In an embodiment of the present invention, a user is provided with the option of selecting a moment from a game from which to create a 3D printed model. In this way, a user can create a tangible souvenir of their in-game experience.

[0071] In an embodiment of the present invention, when the entertainment device receives input indicating that the user wishes to create a 3D print of the current scene, the game state is frozen (eg, paused) so that it can be analyzed for a 3D print.

[0072] It should be understood that for games, many visible features are not just in-game geometry or textures inherent in the model. For example, in-game geometry (which can be a mix of polygonal and skeletal models) can have physics effects such as positioning and deformation applied from one frame to another. Other rendering effects (such as procedural effects and parametric models) can be done in shaders during rendering. At the same time, shaders can also visually process textures based on bump mapping, etc. Therefore, typically when the final scene is rendered for display on the screen, the final scene (as seen by the user) is only assembled in one place, and in one format.

[0073] Therefore, in an embodiment of the present invention, these rendered images are used to construct a 3D model for 3D printing in preference to a potentially different internal in-game representation of the virtual environment geometry. This technique may be referred to as virtual photogrammetry. However, a single rendered image will typically include insufficient information to fully model a virtual environment for 3D printing.

[0074] As previously described, during normal playback, the virtual environment is rendered for a particular viewpoint. Furthermore, in order to achieve high frame rates and efficient processing, then, as part of this rendering process, elements of the environment that are not visible from that particular viewpoint are culled early in the rendering process.

[0075] Therefore, only the elements of the environment that are visible from the selected viewpoint are retained and rendered. If a 3D printed model of the environment were to be generated using this rendering, all aspects of the model that are not visible from that particular viewpoint would be lost or would have to be filled in using some form of interpolation algorithm. Obviously, this would lead to unsatisfactory results when the real printed model is viewed from any other angle.

[0076] Therefore, in an embodiment of the present invention, when the game state is frozen, the entertainment device generates multiple rendered images of the virtual environment from different viewpoints.

[0077] Preferably, at least two opposing viewpoints are rendered, wherein the first viewpoint is typically the viewpoint initially displayed to the user. Between the two images, this results in rendering that culls most, if not all, elements of the environment in the original rendered view.

[0078] Optionally, three viewpoints are rendered, preferably evenly distributed on the plane, with the first viewpoint being the viewpoint initially displayed to the user and the plane being horizontally centered in the view direction of the first viewpoint. Again, this results in rendering that culls most, if not all, environmental elements in the original rendered view, but may capture further environmental elements that may have been occluded by objects in front and behind and therefore not rendered in either of the two viewpoints mentioned above.

[0079] More generally, as more viewpoints are distributed over the plane, fewer elements of the environment remain unrendered. For example, the entertainment device may perform a "fly-by" over the environment, capturing N images, where N is, for example, between 2 and 360. The number of images captured is a tradeoff between the fidelity of the final model and the storage and processing power of the entertainment device required to analyze the captured images (as discussed later in this article).

[0080] Optionally, one or more viewpoints looking down from above the virtual environment may also be rendered (either as separate images or as part of a fly-through sequence) to provide additional detail of those environmental elements that were substantially parallel to a previously captured viewpoint on a single plane (such as the upper surface of certain environmental features).

[0081] In an embodiment of the present invention, metadata related to the position and viewing direction of a virtual camera representing the viewpoint of each image is also recorded and associated with the respective image.

[0082] It should be understood that the virtual environment may represent a large area, while the 3D model will include a limited area determined by the size of the model that can be generated by a particular 3D printer and the scale of the printed environment. Preferably, if a particular 3D printer is in communication with the entertainment device, the characteristics of the particular 3D printer are known (otherwise, a default model size can be assumed or the user can specify the size); similarly, the scale of the printed environment can be selected by the user, or can be automatically determined with reference to a key character within the environment (e.g., the user's avatar). The avatar can be selected to be a specific size within the final 3D printed model (by default, as a non-limiting example, 5 cm high), and the extent of the environment printed at the same scale can therefore be calculated. Alternatively, in a "character printing" mode, only a specific character, such as the user's avatar, can be printed without including surrounding in-game environmental features. This may be particularly valuable for allowing users to customize their own in-game characters and be emotionally dependent on them.

[0083] Thus, an equidistant distribution of two or more viewpoints may be centered about the key character, and any in-game camera control logic may optionally be used to determine the necessary view directions required to center the key character within each captured image.

[0084] In the case of a game presenting a first person view, then based on the model dimensions and the assumed or user-set scale, the center point of the model can be calculated and the viewpoints can be distributed around it.

[0085] In conjunction with the above method, the entertainment device can capture an image by selecting a sampling point on a sphere of a predetermined radius, and the sampling point is centered on the center point. Optionally, those sampling points that are blocked by environmental barriers (such as points on a sphere below the ground or inside a mountain) can be skipped, or the radius of the sphere can be locally reduced until the surface is no longer blocked by the environmental barrier. Alternatively, environmental elements that are not part of the target object to be 3D printed can be omitted from the rendering to provide an unobstructed view (selecting a target object from a wider virtual environment will be described later in this article). Before optionally selecting one or more viewpoints on the sphere above the plane, the sampling scheme can first select a viewpoint on the plane including the original viewpoint and the center point and parallel to the horizontal axis of the original viewpoint, and then optionally select one or more viewpoints on the sphere below the plane. The predetermined radius can be equal to the distance between the original viewpoint and the determined center point of the model to maintain consistency with the original captured image, or the original image and subsequent additional images can be re-rendered at different effective radii, such as the minimum radius at the field of view of the image containing the ground area to be included in the 3D printed model. It is worth noting that games traditionally use lower fidelity models at larger draw distances to simplify the rendering process. Therefore, optionally, the radius can be limited to a maximum distance equal to the render distance, at which the game will select lower fidelity models of key elements of the image (such as the user's avatar). Further optionally, this can be overridden by the user for aesthetic purposes, or because they wish to ensure that a particular environmental feature or combination of features is included in the final model. This can be particularly valuable for allowing users to customize the environment, such as by creating so-called "mods".

[0086] Regardless, the result is a set of images that capture two or more complementary viewpoints of the virtual environment for a given game state.

[0087] It should be understood that the above-mentioned image capture processing can be controlled by the entertainment device. However, alternatively or additionally, the user can capture images of the same scene from one or more different viewpoints by selecting these viewpoints themselves and using traditional image capture processing. These viewpoints may not correspond to the preferred distribution discussed previously in this article. It should also be understood that different users can capture images of the same scene from different viewpoints on different entertainment devices at different times; assuming that the user has access to a pooled set of images (for example, if they are posted to an online forum, or are stills extracted from a "fly-through" video that moves or changes viewpoints, such as may be included in a trailer video for a video game), an equivalent set of two or more complementary viewpoints of the virtual environment can be obtained.

[0088] Given these images and (optionally) associated metadata relating to viewpoint position and orientation, the entertainment device may then proceed to analyze the images to generate 3D model data.

[0089] Several analytical techniques may be used, (optionally) in a complementary manner.

[0090] Contour analysis uses the edges of objects in the captured images to estimate the local object contour of that edge. The object contour can then be extruded (extruded) perpendicular to the contour of each viewpoint until it intersects with another extruded contour (usually extruded from another viewpoint) to create an approximation of the object shape. It should be understood that as the number of viewpoints increases according to the scheme previously described herein, each extrusion will be toward a smaller angle around the object, thereby producing a more accurate overall model of the object.

[0091] Stereo analysis uses the relative displacement of the same objects within two overlapping viewpoints to calculate their distance from the or each viewpoint. From this information, a depth map can be constructed that indicates the 3D shape of the object in the area visible from both viewpoints. Thus, in the same way, where more viewpoints are available, successive pairs of viewpoints can be used to construct a map of the surface of the viewed object.

[0092] In either case, in the absence of viewpoint information, this can be estimated by correlating environmental features between the images, and selecting the viewpoint position and orientation that provides the best correlation of the relative positions of these features.

[0093] It should be understood that contour analysis and stereo analysis can be used to complement each other. For example, the two techniques can be used to detect errors in each other's models; in the case where the two techniques generate estimates of the object surface that differ by a threshold amount, the estimate that most closely matches the interpolation between nearby estimates of the object surface of the two techniques within the threshold agreement can be used, and the other estimates are discarded. Optionally, these areas of the model can also be highlighted for user inspection and / or editing before printing, as discussed later in this article.

[0094] The above technique is particularly useful in situations where the image data is obtained from a third party (e.g., from screenshots on a forum) and no additional metadata is available.

[0095] However, in the case of images generated and captured by the entertainment device, it is preferred to also capture associated depth information generated by the entertainment device itself. As previously described, during traditional viewpoint rendering, environmental elements that are occluded from the rendered view are culled. The occlusion is determined at least in part by the relative distance of objects in the environment to the virtual camera; for example, objects behind other objects are considered to be occluded and therefore culled. For this purpose, these distances are calculated by the entertainment device. This means that for each captured image, there is an accurate per-pixel (or even sub-pixel) resolution distance or "depth" map available.

[0096] In an embodiment of the invention, this depth map is also captured and associated with each image captured by the entertainment device for use in generating the 3D model. In principle, this information can be obtained in a way that is transparent to the operation of the renderer by replicating the so-called z-buffer used when generating the image. Therefore, the image capture process does not require modification of the game rendering engine.

[0097] More generally, it will be appreciated that any suitable technique for identifying the surface position of rendered pixels within each captured image may be made relative to a 3D origin (or, for example, an origin in a game world or for rendering isolated target objects) to generate a consistent representation of the pixel across potentially multiple renderers.

[0098] Reference now Figure 4 , given information about the position A and viewpoint direction B of each image, and associated distance information z, the pixels of each image can be projected to respective points within the shared virtual modeling space 300, for example based on a common origin. In practice, each pixel is displaced by an amount x, y, z from the position A of the image viewpoint in a coordinate scheme local to the image (but which can be rotated to a coordinate scheme common to the shared virtual modeling space using well-known techniques). Here, x and y are the pixel coordinates in the captured image (represented by the abstract screen C), and z is the associated depth or distance value of that pixel from the camera position A. For clarity, in Figure 4 Only one column of pixels (ie, for a single x value) is projected in , but it will be appreciated that any number of pixels of an image may be projected in this manner. For example, pixels corresponding to a target object.

[0099] The result is that pixels from each image will be placed within the shared virtual modeling space, at the correct location on the surface of the virtual objects in the 3D scene, such as Figure 4As shown in the example using Blobman in . When each image is projected into a shared virtual modeling space, more of the surface of each object in the scene will be "painted-in" in this way. In the case where two images project the same pixel onto the same point in 3D space, the second projected pixel can be discarded or the first pixel can be replaced, or an average value can be generated.

[0100] It should be understood that a pixel is a two-dimensional object. Therefore, in an embodiment of the present invention, when a pixel of an image is projected to a position x in the shared virtual modeling space, i ,y j , z k When a voxel (typically a cubic three-dimensional pixel) is actually created at that location, and the color information associated with the projected pixel is used as the color information for that voxel.

[0101] The effect is to build a shell of voxels representing the visible surface of the virtual environment by projecting the color data of the image pixels in each image to an x, y, z position in a shared virtual modeling space.

[0102] It should be understood that instead of using z-buffered depth information from an entertainment device to project multiple images into a shared modeling space, the depth information can be estimated using the stereo depth estimation and / or contour analysis and surface extrusion techniques discussed previously. Thus, these techniques can also be used to drive a collection of voxel shells within a shared modeling space in the manner described above.

[0103] In any case, once all images have been used, the resulting shells of voxels can be analyzed for completeness. Any gaps in the surface (optionally below a threshold size) can be filled by interpolation from the surrounding surface.

[0104] After an estimate of the 3D shape of the environment has been generated in the form of a voxel shell, a 3D model can be generated for printing.

[0105] Determine and improve model integrity

[0106] Reference now Figure 5 In an embodiment of the present invention, a target object (such as the character 520 shown) is rendered from multiple viewpoints. Although it has been suggested that at least two images may be used, or only a small number of images may be used, more images may be used in order to provide good coverage of the target object(s). Figure 5, as a non-limiting example, eighty renderings are generated by the virtual camera for each of the three circular orbits of the character, each orbit being equidistant around a circle orthogonal to the orbit itself. It should be understood that the number of orbits and the number of renderings of each orbit are purely exemplary, and any suitable combination of renderings and orbits (or partial orbits, such as a semicircle above the base height of the target object) may be considered, or other distributions in addition to orbits (such as a distribution of camera positions on the effective surface of the sphere, uniform or non-uniform, for example, with additional camera positions near the position of the in-game camera when the original indication of capturing the object was made). At the same time, for each rendering, depth (distance) data for each rendered pixel from the virtual camera position may optionally also be recorded, so that the position of each rendered pixel in the rendering space can be determined from the rendering, the depth information, and the camera position.

[0107] It will be appreciated that any scheme that results in providing sufficient viewpoints is suitable; thus, instead of three orbiting virtual cameras, a cluster of fixed virtual cameras (e.g., 80) may be used. Similarly, in either case, the target object may be static or itself may rotate on one or more axes, for example to provide several adjacent views for each of the smaller camera clusters, effectively compensating for missing cameras at different viewpoints in the larger cluster.

[0108] The resulting N images (240 in this example) are provided as input to known photogrammetry techniques, such as described above.

[0109] Example output of this technique is in Fig. 6A and 6B Shown in. Fig. 6A A model of the character derived using virtual photogrammetry is shown and appears to be accurate on the surface. Figure 6B It can be seen that the area of ​​the character's torso that is primarily occluded by the gun is not accurately modeled, as highlighted by the elliptical ring, resulting in areas of the model being blended or deformed.

[0110] Obviously, users who have printed such a model will be disappointed with the quality of the result.

[0111] Therefore, we now also refer to Figure 7 In an embodiment of the present invention, the 3D printing modeling method includes, in a first step S710, generating a voxel representation of a surface of a target virtual object.

[0112] It should be understood that by freezing the virtual environment / target object for multiple renderings, the actual object model remains static during the virtual photogrammetry process. Therefore, a separate and persistent representation of the model can be generated to track the progress of data collected by each consecutive rendering. Fig. 8A and8B ,for Fig. 8A In the example case of a samurai character rendered in , you can generate Figure 8B A representation of the character shown (in this case in a slightly different pose, about to swing the axe).

[0113] A single and persistent representation is a voxel model of the surface of the target object to be printed. Fig. 8A and 8B It can be seen from the comparison that usually the voxel resolution is lower than the rendering resolution, so that multiple pixels in the rendered surface will be mapped to corresponding voxels in the voxel model.

[0114] Optionally, the voxel resolution can be selected to correspond to the voxel resolution of the 3D printer to be used. So, for example, if the 3D printer has a 1mm 3 If the voxel resolution of the 3D printer is 10 cm high, and the printed model will be 10 cm high, the voxel model will be 100 voxels high. It should be understood that while it is preferred to use at least the same or higher voxel resolution as the 3D printer, this is not required.

[0115] The voxel model may be generated using model data extracted during the rendering process, or the voxel model may be rendered separately using the same target object model data.

[0116] In a second step S720, a counter is associated with each voxel. The counter is used to count how many rendered images include pixels from the portion of the target object(s) corresponding to the voxel, as will be explained later herein. Thus, the voxels together with their counters form a voxel histogram, which represents how many times each surface area corresponding to the voxel was rendered during the photogrammetric processing. Thus, the counter can be considered as a master histogram counter for the voxel.

[0117] In the third step S730, as described above, the target virtual object(s) are rendered from multiple viewpoints, for example, referring to Figure 5 .

[0118] Then, for each rendering viewpoint, in a fourth step S740, a counter associated with a voxel is incremented if the first pixel on the object surface portion corresponding to the voxel is rendered.

[0119] As mentioned before, typically the voxel resolution is lower than the render pixel resolution, which means that potentially multiple (and variable) numbers of pixels may map to corresponding voxels in any given render. Therefore, the total number of pixels corresponding to voxels in a render will be uninformative, and if the counts are accumulated over multiple renders this will be mixed.

[0120] Rather, it may be assumed that the first pixel in a rendering that maps back to a voxel may be considered to indicate that the region of the model corresponding to the voxel is appropriately visibly represented in the rendering.

[0121] A simple way to implement this checking method is to further associate a flag with each voxel, wherein each voxel is reset to a first state for each rendering (e.g., to inactive, low, or 0, depending on convention). Subsequently, the counter of the voxel is only incremented if the previously described conditions are met for rendering a pixel on the object surface portion corresponding to that voxel, and the condition that its associated flag is in the first state. Then, if the main histogram counter is incremented, the flag is also switched to the second state (e.g., to active, high, or 1, depending on convention).

[0122] The main histogram counter may be incremented only once per render, i.e. in response to detecting that the first rendered pixel is located in a region of the surface of the target object(s) corresponding to that voxel, by flipping the flag to a different state, and only incrementing the counter if the flag is in the default initial state.

[0123] It will be appreciated that in principle different counting strategies may be used in addition to or instead of the flag scheme. For example, a parallel counter may optionally be provided for each voxel, which may count to M when a pixel corresponding to that voxel appears in the rendering (where, for example, M is a number corresponding to the fraction of pixels that in principle map to the voxel).

[0124] So, for example, in the previous 1mm for the 10cm model 3 In the voxel case, this results in a voxel representation that is 100 voxels high, and then if the rendered object is 1000 pixels high, it can be inferred that perhaps 100 pixels (10×10 pixels) can be mapped onto the voxel, for example if viewed from the front and without occlusion.

[0125] Thus, for 1 / 2 of the pixels, the parallel counter will count to M=50. Meanwhile for 1 / 25 of the pixels, the parallel counter will count to M=4. In this case, the main histogram counter may be incremented only when the parallel counter reaches the selected M threshold. This will indicate that a significant proportion of pixels corresponding to that voxel have been rendered in a given view. This proportion may be, for example, anywhere in the range of 100% to 1%, but more typically in the range of 50%-1%, more typically still in the range of 25%-2%, more typically still in the range of 15%-3%, and more typically still in the range of 10%-4%.

[0126] Similarly optionally, a parallel counter can be provided that counts to P, where P is again the proportion of possible pixels that can be rendered. At the end of the analysis of a given rendering, the final value in the parallel counter is added to the main histogram counter. This will allow reflections of different numbers of pixels to be rendered for each voxel in the view, while limiting the inconsistency between the histogram values. Continuing the above example, if the fraction is 1 / 10, so that P=10, then a voxel that has 100 corresponding pixels rendered will only have 10 added to its histogram, while a voxel that has 5 corresponding pixels rendered will have all 5 added to its histogram. Over time, it will still be obvious which areas of the model perform better than others, but the difference is unlikely to reach multiple orders of magnitude, which may make subsequent processing difficult.

[0127] Thus, in this case, the method will comprise associating each voxel with a second counter limited to a predetermined maximum count value, and the step of increasing the main histogram counter will comprise increasing the second counter associated with the voxel, which is subject to the predetermined maximum count value, and then increasing the main histogram counter by the final value of the second counter if a pixel on the portion of the object surface corresponding to the voxel is rendered.

[0128] Refer again Figure 5 and Figure 8B Once all renderings at predetermined camera positions have been completed and the relevant data for photogrammetry has been accumulated, the master histogram counters for the individual voxel models will have values ​​indicating how often (or, alternatively, how completely) the corresponding area of ​​the target object was rendered at all different camera positions.

[0129] Then, now refer to Fig. 9 In an embodiment of the present invention, for each voxel, the 3D printing modeling method is performed as follows:

[0130] In a fifth step S950, the counter is compared to a predetermined threshold to detect whether it is below the value. If it is above the threshold, it is assumed that the region of the target object(s) corresponding to the voxel has been imaged frequently enough to provide a good input to the photogrammetric processing, which may be performed as previously described in order to generate 3D surface geometry (and optionally but preferably also associated textures). The generated 3D surface geometry may be in the form of polygons, point clouds, high-resolution voxel shells, or any suitable representation for subsequent processing as described later herein.

[0131] However, if the counter is below a threshold, it is assumed that the region of the target object(s) corresponding to that voxel is not imaged frequently enough to provide good input to the photogrammetric processing, and therefore at least one additional rendering is needed. For example, a QR rendering may be sought, where Q is a predetermined threshold and R is the actual value of the main histogram counter for that voxel. Alternatively, only one additional rendering may be sought.

[0132] To obtain this rendering, in a sixth step s960, a plurality of lines are projected from the position of the voxel (e.g., 16, 32, 64, 128, 256 lines radiating equidistantly from the center or surface of the voxel). These lines can be considered as candidate sight lines for the virtual camera, which are guaranteed to intersect the region of the target object(s) corresponding to the voxel (when looking down the line towards the voxel position).

[0133] Then, a seventh step s970 comprises detecting whether any line reaches a predetermined distance without being occluded by other voxels. Figure 5 , you can detect if any line reaches the surface of the sphere where the virtual camera is without being interrupted / occluded by other voxels.

[0134] It will be appreciated that if the camera is looking along isoequivalent lines, the intersection of any line with the sphere represents where a virtual camera may be placed to obtain a rendering that includes at least some area of ​​the target object(s) corresponding to the voxels.

[0135] Therefore, in an eighth step s980, at least one such line is selected (or transposed to the rendering space if the rendering space uses different coordinates) as an optical axis for further rendering (multiple) target virtual objects, and in step s990, a rendering of (multiple) target objects along the selected optical axis or each selected optical axis is then generated.

[0136] In case a plurality of lines reach a predetermined distance without interruption, then optionally all of them, or a predetermined number of them, or a QR of them, or just one may be used in this manner. In case a subset of the available lines is used, optionally the lines are selected to maximize the distance between the selected lines on the camera sphere in order to obtain different viewpoints of the relevant area of ​​the target model(s).

[0137] In this manner, additional renderings that capture views of under-represented areas of the target model may be identified and obtained in order to improve the desired coverage of the model for photogrammetry purposes.

[0138] Thus, after this processing has been completed for any voxels whose counts are below a predetermined threshold, again, rendering can provide good input for photogrammetric processing, which can be performed as previously described to generate 3D surface geometry (and optionally but preferably, associated textures). Again, the generated 3D surface geometry can be in the form of polygons, point clouds, high-resolution voxel shells, or any suitable representation for subsequent processing as described later herein.

[0139] Therefore, whether or not additional rendering is required, the direct product of the photogrammetric processing is the 3D surface geometry of the target object (and optionally any associated textures). This 3D surface geometry (and optionally textures) can be exported to a separate device for modification (as described later in this article), or saved and shared to a local or remote private or public repository.

[0140] At the same time, if no line reaches a predetermined distance for a given voxel without being occluded by other voxels, the number of lines can be optionally increased (e.g., from 128 to 256 lines), thereby increasing the line density and the chance that the line has an uninterrupted path. Such processing can continue until the maximum number of lines is reached (e.g., 512).

[0141] If no line reaches the predetermined distance without occlusion (optionally after increasing the number of lines), the method may include the step of warning the user that the resulting photogrammetry will be based on an incomplete view of the target object(s) and that a 3D printed model based on the virtual photogrammetry of the rendered object may contain errors. The method may then terminate, providing the user with an option to terminate, providing the user with a choice to proceed or continue.

[0142] In particular, where the number of lines reaches a maximum number without any line reaching the predetermined distance, this means that the view of the model area corresponding to the voxel is very limited and therefore any resulting distortion or error in the model in this area is likely to have little aesthetic impact.

[0143] Optionally, the user may be provided with one or more renderings of the model overlaid with a so-called "heat map" that color codes regions of the model based on the histogram counter values ​​of the corresponding voxels. This provides the user with a simple indication of the quality of the source data (e.g., the number of renders) for any given region of the render.

[0144] Similarly, after the photogrammetry is complete, the resulting model can be rendered for the user so that, for example, the user can control the viewpoint and inspect the model. Also optionally, the resulting model can provide a color heat map overlay that indicates to the user the quality of the source data (e.g., number of renders) for any given area of ​​the resulting model. This can help the user more closely inspect areas of the model that may include errors before submitting the model for 3D printing.

[0145] Modify the model

[0146] It should be understood that the 3D surface geometry generated by photogrammetric processing, whether represented in polygons, or as a point cloud or as a high-resolution voxel shell, will have a negligible thickness, which corresponds to an abstract thickness of zero or one pixel thickness or one (high-resolution) voxel thickness. In any case, when transmitted to a 3D printer, this is likely to be interpreted as one printing voxel thickness (e.g., using the example above, 1mm thickness). For any given model, this is unlikely to be sufficient in practice.

[0147] Thus, now considering the 3D surface geometry as a shell of voxels to be printed, the shell can be extruded to a predetermined thickness. The extrusion can be performed by adding adjacent voxels to each surface voxel on the opposite side of the side from which they are projected into the shared virtual modeling space (i.e., on the inside or underside of the shell). The number of voxels added can be the smaller of Y voxels or the number required to reach the opposite existing surface of the object. Y can be selected to result in a thickness responsive to the weight and strength of the material used to print the model.

[0148] Similarly, the lowest point in the (now extruded) model may be determined. At least two additional points (typically on the underside of the shell) may be further extruded to match the lowest point, thereby providing at least three support points for the physical model. Preferably, these support points are distributed around the calculated centroid of the model.

[0149] In this way, a 3D model of a virtual environment can be built from images from multiple games, which persists when physically printed and can be viewed from multiple angles.

[0150] The model is then sent to the 3D printer driver, which cuts the model from the bottom up into multiple layers. These layers are then printed consecutively by the 3D printer as described above.

[0151] As described above, photogrammetry of at least a portion of a virtual environment can be used to generate a voxel shell that can be used as the basis for a 3D printed model. Some environments are more suitable for this 3D printing process than others. For example, The simple block-based environment within is perfect for rendering via a 3D printer.

[0152] However, this is the exception rather than the rule. The particular appeal of video games lies in their ability to present environments and characters that are not subject to the normal rules of physics. In particular, objects can exist in predetermined relationships to each other without necessarily being physically connected (e.g. Figure 3 , where his hands and feet are not physically attached to his torso), while other objects can be defined in only two dimensions within a three-dimensional environment, such as curtains, capes, and in many cases environmental components such as walls. This is because physical strength is not a requirement for virtual environments, where procedural rules prevent movement beyond certain boundaries, and walls are merely decorative surfaces with textures used to demarcate those boundaries.

[0153] Thus, the virtual environment may include several boundary surfaces with zero thickness on which physical objects are placed, some of which may otherwise be unstable or unsupported if they were to be physically printed. Examples of such boundary surfaces may be Figure 3 The walls of the room 210 in the virtual environment 200.

[0154] It may not be feasible to credibly print such a virtual environment using a 3D printer.

[0155] Therefore, in an embodiment of the present invention, the previously described voxel shells are modified to take into account the physical requirements of 3D printed models.

[0156] As previously noted, the shell of a voxel can be extruded to M voxels thick to provide some physical strength. This also solves the problem of 2D objects in the environment, since the shells of the voxels corresponding to these objects are extruded in the same way as any other element of the voxel's shell.

[0157] However, for some elements of the environment this may not provide sufficient physical support, or in the case of aerial objects in the environment, support may be completely lacking.

[0158] Procedural complementary geometry

[0159] Thus, procedurally defined modifications to voxel shells can be provided for 3D printing purposes. These procedural modifications provide structural strength and / or physical support to some elements of the virtual environment that would otherwise be infeasible to replicate in a 3D printed model.

[0160] In the following, references to "printer geometry" refer to the voxels created to define the 3D model for printing.

[0161] Once the initial voxel shell has been generated, the entertainment device may generate procedurally generated or modified printer geometry.

[0162] The voxel shells are analyzed to detect whether additional geometry is needed based on a few basic rules.

[0163] One analysis step is to detect whether there are unsupported objects in the environment.

[0164] If so, these may be subject to a minimum size threshold so that objects below that threshold (such as, for example, snowflakes, bullets, etc.) are ignored and subsequently not retained in the 3D model. The associated voxels in the voxel shell may be deleted. Optionally, this step may be implemented in a modified rendering process so that when images are collected for the model, these items are not rendered in the first place, thereby simplifying analysis of the images.

[0165] Meanwhile, for objects that meet the threshold, printer geometry of support structures such as connecting rods can be procedurally generated between the object and a nearby anchor point, which is part of the environment that is ultimately connected to the ground. The ground itself can be assumed to be connected to the lowest point of the voxel shell. Optionally, the analysis can be performed from the lowest part of the voxel shell upwards, so that unsupported objects can be supported sequentially, enabling them to support subsequent objects in turn.

[0166] So, for example, when a video game character jumps into the air, by creating a voxel cylinder (e.g., M voxels thick by default), a support pole will be formed between the soles of its feet and the ground.

[0167] However, the thickness of the rod can also be determined programmatically based on basic rules. For a given type of printer, the volume weight of common 3D printer materials and their compression and bending strengths will be known. In addition, the expected size of the printed model will also be known. Therefore, the entertainment device can estimate the weight of the unsupported object and calculate the thickness of the support rod required to fully support the object in the model. Specifically, in the case where the connecting rod is partially or completely transverse and may therefore bend, the moment of the unsupported object is a function of its weight multiplied by its distance from its current position along the rod. In addition, when constructing a connecting rod for such an object, the weight of the connecting rod itself may account for a large proportion of the weight. Therefore, since the various parts of the rod support a greater weight at the anchor point than at the unsupported object, the rod may be thicker closer to the anchor point and thinner towards the unsupported object.

[0168] In another analysis step, the above principles can be applied to other parts of the voxel shells; for example, a virtual model tabletop in a game may have legs that are too thin to support the combined mass of the tabletop and any objects on it. For example, if a character is standing on the table, but the legs are only a few voxels thick, they may not be strong enough to support the physical model. Therefore, the characteristics of the voxel shells can be evaluated to determine the total weight they support in the model, and in the event that this exceeds the safe limit of the cross-sectional area of ​​the voxel shell supporting this weight, this can be thickened by enlarging its cross-sectional area and filling the voxels. For example, this principle can also be applied to trees in the environment, where the accumulated weight of the branches and trunk is used to locally modify the geometry of the branches or trunk to make their cross-section larger to accommodate the weight, in which case the existing geometry is locally insufficient to accommodate the weight when rendered as a 3D model using special materials associated with a 3D printer.

[0169] The center of gravity of the adjusted model can then be calculated to detect if it will tip over. If so, the supports can be moved to reestablish stability, or alternatively, the voxel shell can be thickened at one or more selected locations to change the center of gravity back to a stable position.

[0170] As previously described, the two-dimensional components of the environment are extruded along with the rest of the initial voxel shell to generate a three-dimensional object having at least a minimum thickness. However, this extrusion process may be further modified as described below.

[0171] Now refer to FIG. 10A to FIG. 10E , it can be seen that Fig. 10A A two-dimensional wall from environment 200 is shown, on which in-game candles and candlesticks are placed. Then, Fig. 10B The extrusion process is described to extend the wall in a direction perpendicular to the plane of the wall and opposite to the image projection side to procedurally generate a geometric shape, such as a Y voxel, describing the wall with a predetermined thickness. Fig. 10C As shown, the cumulative weight of a physically printed wall will vary depending on the location on the wall, and some parts of the wall that additionally support the candles and candlesticks will also have step changes. Therefore, walls of uniform thickness may not be appropriate because the wall thickness close to the base may not be sufficient to fully support the cumulative weight of the model.

[0172] Therefore, refer to FIG. 10D to FIG. 10E , then starting at a minimum preferred extrusion thickness at the top of the wall, the extrusion thickness increases as a function of the accumulated weight, resulting in a step change in thickness at the attachment point of the candlestick to the wall to provide additional support. It should be understood that FIG. 10D to FIG. 10EAs shown, the procedural rules may thicken the area around such attachment points, thereby thickening the wall slightly before the load of the candlestick is applied to the wall. In addition, the procedural rules can accommodate the additional mass of the support thickness itself when squeezing the lower part of the wall. Finally, the load applied by the candlestick and the support area can be distributed both laterally and longitudinally, so that the thickened support area spreads out laterally as it propagates down the wall, as shown in Fig.10E shown.

[0173] Procedural rules can also interact with additional information provided by the game developer to help automatically generate the required auxiliary geometry.

[0174] For example, the aforementioned rule, where links are procedurally generated between unsupported objects and the nearest anchor point, where the anchor point is the part of the environment that is ultimately connected to the ground, can result in Fig.11 3D model of Blobman 220' shown. It should be understood that when the character's hands are analyzed, it is likely that they are closer to the character's feet than the character's torso, and the character's torso may not yet be connected to the anchor structure itself. Accordingly, the procedural rules may attach the character's hands to the only available grounded anchor points (i.e., the feet); the torso may then be anchored to the nearest single anchor point (now a hand). The result is a structure that, while stable, does not anatomically conform to the user's expectations.

[0175] Thus, a game developer (or user) can identify preferred connection points between environmental elements to guide the procedural generation of linkages or other connection / support structures. These points of connection can be defined using reserved values ​​within one or more color channels, so that connection points can be identified using specially colored identifying pixels within the image; for example, if a value of 128 is reserved on the blue channel to represent a connection point, then Fig.12 The Blobman character 220 example uses this value to specify preferred connection points that exist on the model. These reserved values ​​are then used in the voxel shells, creating identifiable anchor points for use with procedural rules. For example, pairs of connection points can be indicated using matching values ​​in the other two color channels of the identification pixels, so that the red and green values ​​in the identification pixels are the same for the hands and torso, making these identified parts connected together. In this way, the developer can maintain aesthetic control over the placement of connected structures without the need for separate metadata. In the case of Blobman, this can be used to ensure a set of anatomically acceptable links, such as Figure 7 At the same time, the procedural generation rules for the connection / support structures will still determine the cross-sections required to adequately support the printed structure.

[0176] It should also be understood that for elements that supplement the printer geometry that are not found in the rendered game (such as the links in the "Blobman" example above), the texture (color) can be inferred from the existing voxel shells, or no color can be assigned. In the case of inferred color, the color can optionally be inferred from the part of the structure that supports the element against gravity (e.g., the ground, or in the case of Blobman, from the feet to the torso, and from the torso to the hands and head, respectively), as this may create the least obtrusive support structure in the model. Alternatively, in the case of no texture, the bare plastic (or other 3D printer material) is exposed, making it clear to the viewer that this is a functional support and not part of the original image. It may be an aesthetic decision for the user which approach is preferred.

[0177] It should also be understood that, at the scale selected for printing, the pixel resolution of the rendered image may be higher than the voxel resolution of the 3D printer. Therefore, features of the rendered environment may sometimes have high spatial frequencies relative to the voxel resolution of the 3D printer. Therefore, the image may be filtered to a compatible resolution before being projected into the shared virtual model space to remove such high frequency details and thereby avoid aliasing effects in the resulting model. In this case, if the programmatic rules are to use the preserved color channel values, then these values ​​are preserved or re-inserted into the lower resolution image after filtering. Depth information may be similarly filtered.

[0178] Thus, using any or the above techniques or any other suitable techniques, the properties of a 3D surface geometry (such as its thickness or connecting elements) may be modified to form 3D printable model data that may be output for 3D printing immediately or prospectively in the future. Thus, the direct product of modifying the properties of a 3D surface geometry to form subsequently output 3D printable model data is the 3D printable model data. Prior to any decision to print the model, the 3D printable model data may be output directly to a printer, or alternatively to a local or remote private or public repository for viewing by a user or others.

[0179] Flood Fill Algorithm

[0180] It should be understood that when 3D printed models come from a computer-aided design package, these models are well structured and easy to process for 3D printing. However, when video game data is to be printed, the captured model may contain degenerate polygons and / or zero thickness features (such as those discussed earlier in this document in connection with FIG. 10).

[0181] One way to make these models printable is to first identify the solid (closed) areas in the model, and then perform operations to strengthen or thicken these areas as needed, for example according to methods such as described elsewhere herein.

[0182] Now refer to FIG. 13A to FIG. 13C In order to identify solid or closed areas, such as within a slice of a 3D voxel model, a flood filling technique can be used to identify solid areas in the slice.

[0183] Fig.13A An example 2D slice is shown, in this case including five closed regions. To identify these regions, then Fig. 13B In , a flood fill can be initiated from the outer boundary of the patch to occupy those parts of the patch that are not in the enclosed area.

[0184] The resulting flood fill can be considered as a mask, and the enclosed regions are identified by inverting the mask, as Fig. 13C shown.

[0185] While in principle it is possible to fill each of the individual closed regions directly, which may require less computational resources if the area of ​​the patch being filled is smaller, this requires foreknowledge of the locations of the closed regions. Such foreknowledge cannot be guaranteed. Therefore, by using flood fill from outside the patch, closed regions can be identified without any prior knowledge or assumptions about their existence or location.

[0186] However, it should be understood that for detailed 3-D models, the number of tiles processed in this manner may be very large, and each tile may have a high resolution. As a result, the computational load of the flood fill process may be very high. This problem may be exacerbated if the fill process is used frequently, for example, to evaluate the integrity of the model after or during the creation of supplementary geometry, or after user edits have been made.

[0187] Therefore, it is preferable to implement a computationally efficient flood-fill algorithm.

[0188] The traditional flood fill algorithm is a popular method to fill connected areas of pixels of the same color with another color. However, this algorithm is sequential in nature as the algorithm fills them step by step, pixel by pixel.

[0189] Meanwhile, the preferred processing platform for this flood fill algorithm is the GPU on entertainment devices such as the Sony PlayStation 4. However, the GPU has a large number of running threads that cannot efficiently run sequential processes.

[0190] Therefore, in an embodiment of the present invention, in order to utilize GPU processing, a parallel flood fill algorithm is provided so that part of the virtual processing can run simultaneously.

[0191] In short, the parallel flood fill algorithm constructs a quadtree for the image to be filled. The quadtree stores a multi-resolution representation of the flood fill source pixels as well as some additional data (described below) to speed up the fill operation.

[0192] The quadtree traversal starts at the root of the tree (for outside-in filling) or at a leaf node of the tree (for pixel seed filling). The traversal is performed at the lowest quadtree level (largest area node) allowed by the size of local features in the image to increase the speed of filling across the image.

[0193] During traversal, the entry boundaries of each quadtree cell processed are tracked to correctly traverse to child quadtree nodes and guide flooding operations when stepping to adjacent cells.

[0194] The result of the algorithm is a flood mask, which indicates the pixels that have been filled. The mask is preferably stored in a Morton (z-order) curve so that all pixels associated with a quadtree cell (at any quadtree level) are contiguous for faster processing. The mask can be later used to generate an image from a source image with flooded pixels, or for other purposes, as described later in this document.

[0195] It is worth noting that the algorithm can also be extended to three dimensions to implement volumetric flood filling by using an octree, keeping track of all 8 inlet planes for each cell, and a 3-component Morton curve that fills the resulting mask addresses.

[0196] In more detail, the algorithm is described below using the following terminology:

[0197] - Flood Fill Source Color: This is the source color in the image to be replaced by the target flood fill color. For example, the flood fill source color in Image 13A and Image 14 is black.

[0198] - Flood Fill Destination Color: This is the destination color of the connected areas that replace the flood fill source color. For example, the fill destination color can be white.

[0199] -Boundary pixels: These are any pixels in the source image that are not flood-filled with the source color. Fig.14 In , the boundary pixels form three spirals and are primarily monochrome, but the antialiased border that blends that color with the flood fill source color is also a boundary pixel. Therefore, all boundary pixels are typically preserved during flood filling (optionally within a predetermined tolerance of the flood fill source color).

[0200] - Quadtree: An acceleration structure (spatial partitioning scheme) used for multi-threaded traversal over an image at different resolutions based on feature details.

[0201] In an embodiment of the present invention, the quadtree used to process flood fill contains 3 pieces of information:

[0202] 1) Bit ("occupied_all") indicates that all pixels within the quadtree cell are boundaries (non-floodfill source color).

[0203] 2) A bit ("occupied_some") indicates that at least one pixel within the quadtree cell is a boundary (non-floodfill source color).

[0204] 3) For cells with no border pixels at all, store an elevation depth that indicates how much the quadtree can climb towards the root while still remaining in a completely non-border cell.

[0205] For quadtree entries, 6 bits are typically needed, the elevation depth is capped at log2(image size) for image sizes up to 65536x65536, and one bit is needed for each of the other two fields.

[0206] Elevation depth is a good indicator or representation of the visibility of the flood fill source color at any point in the image. Fig.15 Shows Fig.14 The elevation depth of an image and the resulting quadtree structure are shown, where lighter grey shades represent greater elevation depths, meaning that larger quadtree cells will be used to fill that area of ​​the image.

[0207] For local feature sizes, the flood fill algorithm uses the quadtree level closest to the root of the quadtree. Therefore, as the fill operation approaches the boundary, it will move toward the leaves of the tree. If the algorithm squeezes into open areas through gaps in the filled area, it will move back toward the root of the quadtree to fill more efficiently.

[0208] In the first step, the top-level pixel occupancies are generated.

[0209] It identifies occupancy information for boundary pixels in the image by testing the flood source color. The result is a 2D array of quadtree cells that are the leaves of the quadtree, where each cell is associated with a pixel in the source image and contains the same value in the "occupied_all" and "occupied_some" bits (since each cell contains 1 pixel).

[0210] In the second step, the mipmap layer pixel occupancy is generated.

[0211] For each 2×2 cell group at the occupancy level, the occupied_total and occupied_some information of the merged cell is generated by ANDing all sub-occupancy_total and ORing all sub-occupancy_some fields. This process is repeated until there is only 1 cell at the top level, which contains the occupied_total and occupied_some of the entire image. After this stage, the quadtree structure is complete. The next stage generates some additional information to speed up the traversal process during runtime.

[0212] At any point before, during, or after the first or second step above, the cell elevation depth is also initialized to zero.

[0213] In the third step, depth elevation values ​​are generated.

[0214] Starting at the root of the quadtree, and working towards the leaves, for all cells that have no border pixels at all (occupy_some=0), the elevation depth of its 4 children is set to 1 greater than the current cell. This is repeated all the way to the leaf nodes of the quadtree (each node represents 1 pixel in the image). For any cell, the elevation depth allows the algorithm to move towards the root of the tree while staying within an empty node, and allows the algorithm to quickly process large, contiguous areas of flood-filled source color.

[0215] Fig.16 A slice through a quadtree (represented here as a binary tree) is shown, showing possible stored data. The bottom row includes 16 pixels of an example source image, combined into a single node at the root of the tree. In each cell, the first number is the elevation depth, and describes how far up the tree you can go while remaining in a completely empty cell. So, in this case, the 5 topmost cells (1610-50) can be identified.

[0216] In the fourth step, a flood fill is performed.

[0217] Flood fill can be run in two modes. Outside-in or start from a seed pixel. The two methods differ only in the starting condition of the algorithm. Outside-in starts from the outside of the image, starting from the root of the quadtree (where a quadtree cell represents the entire image). Pixel seed starts from a leaf node of the quadtree (where a node represents a single pixel).

[0218] The algorithm works with a bunch of work-items that can run in parallel. A work-list item identifies a node in the quadtree, and a list of directions from it into the cell. These directions allow the algorithm to make decisions when moving down the quadtree to a finer cell, or when jumping to a neighboring cell. The format of a work-list item is:

[0219] Quadtree Hierarchy The list of boundaries through which the cell is entered Unit X Unit Y

[0220] It should be appreciated that the flood fill algorithm effectively has a flow as it progresses through the image / cell of the tree structure. The entry boundary of the cell tracks the direction of the flood fill as it flows through the image / cell.

[0221] A cell may be entered from zero or more boundaries (zero only occurs for the initial seed fill pixel, all 4 boundaries only occur for the initial "whole image" cell filled from the outside in). As the algorithm progresses, cells are typically entered through one entry boundary (a side) or two entry boundaries (e.g., a corner).

[0222] A cell has 4 sides, with the top, bottom, left, and right sides defining the quadrilateral. The algorithm then keeps track of how the cell was entered so that subcells can be selected in the correct order, as explained below with respect to subcell recursion. This occurs when a cell contains border pixels that may prevent flood filling. Using the border to select subcells ensures that subcells are processed in the correct order and prevents flood filling from skipping border features.

[0223] It should be appreciated that for an octree there are 6 boundaries (faces of the cube), but the principles are otherwise the same.

[0224] Push work list items onto a stack from multiple threads in a GPU thread group.

[0225] The initial state of a worklist contains a single workitem, depending on the fill mode.

[0226] For outside to inside:

[0227] Quadtree The list of boundaries through which the cell is entered Unit X Unit Y Root level Left, Right, Up, Down 0 0

[0228] For pixel seed:

[0229] Quadtree The list of boundaries through which the cell is entered Unit X Unit Y Leaf Level NA Pixel X Pixel Y

[0230] As mentioned above, many work items can be popped from the stack and processed in parallel by many GPU threads when the algorithm proceeds as follows: Get a work item from the stack, which specifies the following information:

[0231] Quadtree hierarchy.

[0232] The entry boundary of the cell.

[0233] Cell XY coordinates.

[0234] Read the occupancy of a work item unit. The occupancy specifies the following information:

[0235] Elevation depth.

[0236] Occupied_All bits indicate that all pixels in the cell are boundaries.

[0237] Occupies some bits, indicating that some pixels in the cell are boundaries.

[0238] The cell is then raised using the occupied elevation depth. This requires:

[0239] Adjust the quadtree level from the work item using the occupied elevation depth.

[0240] Adjust cell coordinates using elevation depth.

[0241] The work that needs to be generated for the cell is then determined based on occupied_all_bits and occupied_some_bits:

[0242] Occupancy_All Occupy_some operate 1 Any Discard the unit without further action 0 1 Recurse to subunits based on entry boundaries 0 0 Fill the cell and traverse the neighbors based on the entry boundary

[0243] The process then repeats from the beginning until there are no more work items to process.

[0244] The above three operations (unit processing scenario) are as follows:

[0245] "Discard cell, no further action": In this case, the cell is completely filled with the border color, no further processing is done on this cell, and it can be discarded. No further work is generated for the work item stack.

[0246] "Recurse into subunits based on entry boundaries": Reference Fig.17A , the cell is partially filled and may contain some space to be filled. Work items are generated for the child cells, selected by the entry boundaries. For example, entering the parent cell from the top and left will result in three child cells being processed, with Fig.17A The various combinations or entry flags described in (entry boundaries indicated by thick edges). The new work item is pushed onto the stacks of all related subunits using the correct boundary entry flags.

[0247] "Fill cells and traverse neighbors based on entry boundaries": Reference Fig. 17B , in which case the border pixels of the cell are completely empty and need to be filled. However, since there are multiple paths to the cell, the cell may have already been filled by another thread / work-item. If the cell has not been filled (determined by testing the fill mask result), the fill should traverse to the neighbors to avoid the neighbors from which it was entered. The new work-item is pushed onto the stack of all relevant neighbor cells with the correct boundary entry flags. It should be understood that the entry boundary is the boundary from which the flood fill enters the cell (such as Fig. 17B ), and when traversing to the neighbors, the output unit is all adjacent units except the entry boundary unit, as shown in Fig. 17B As shown in the right image of , it can be assumed that the cells in the direction entering the current cell have already been processed.

[0248] Using this method, the flood fill algorithm builds the fill mask, and the quadtree elevation mechanism ensures that the cells filled are as large as possible based on the local feature size. If the fill mask is stored in Morton order, all pixels of the quadtree cell are contiguous in the fill mask and processing is faster.

[0249] The quadtree approach above provides an efficient computational implementation of flood fill when using a GPU or similar parallel processor with a large number of threads that cannot efficiently run a sequential process. It can analyze large numbers of slices of a model that may be 3D printed, to analyze completely enclosed areas, etc.

[0250] Furthermore, in some cases, the algorithm performs better than the traditional flood-fill algorithm; see Fig.18A , which replicates Fig.13A but now the objects within the patch extend to the borders of the image (or in other words, the image is cropped to the smallest size that contains the object within it, which of course reduces the overall image size and thereby reduces computational complexity).

[0251] However, now refer to Fig.18B and 18C , the traditional flood filling starting from the initial point (e.g., the upper left corner of the image) cannot enter the lower left and lower right corners of the image, as shown in Fig.18B As shown, these are effectively blocked by objects in the image. Fig. 18C The enclosed area is incorrectly described.

[0252] In contrast, the present invention can fill based on patch / image boundaries rather than a single seed point (although this can also be done, as described above); as a result it can still correctly identify the regions in the lower left and right corners of the image.

[0253] It should be understood that this is also due to Fig.14 and Fig.15 Due to the extent of the spiral pattern, it actually divides the background into four separate parts; however, the system recognizes each of them (which can be seen from Fig.15 inferred from the elevation map in ).

[0254] However, even the quadtree algorithm can run into difficulties, so there is room for further refinement:

[0255] Reference Fig.19A In the example scene, a pair of cylinders (1910, 1920) are concentrically arranged within an object in the game environment for 3D printing. Fig.19B , the resulting slice through the object shows concentric walls defining alternating voids and closed areas within the slice.

[0256] Reference Fig.19C , even when filling from the boundaries of the patch image, the flood fill algorithm cannot (and should not) pass through the outer wall of the outer cylinder. As a result, the intermediate gap between the outer cylinder and the inner cylinder and the central gap of the inner cylinder are not identified.

[0257] As a result, Fig.19D In the 3D printing process, the internal structure of the cylinder is lost and it is represented as a solid object. Obviously, this is undesirable both in terms of loss of fidelity of the model and in terms of waste of 3D printing material.

[0258] Therefore, in an embodiment of the present invention, the use of a quadtree to navigate 2D image slices is replaced by an octree to navigate a 3D (volumetric) image.

[0259] In other words, instead of using a quadtree to perform a flood fill on a sequence of 2D slices of the model, the algorithm uses an octree to perform a flood fill on a single stereo image of the model. It should be understood that each level in the quadtree divides a rectangle into four sub-rectangles, while each level in the octree divides a cuboid into eight sub-cubic blocks. The rest of the algorithm related to creating work item elevation depth, occupancy data, etc. is essentially the same (the volume cell has six possible entry planes to achieve the same function as the four possible entry boundaries of a 2D cell).

[0260] Back to Fig.19A As an example, it will be appreciated that when represented by a single stereo image and volumetric flood filling is performed using the octree version of the algorithm described previously, the problem of concentric cylinders is solved because the algorithm can identify the gaps at the top and bottom of the cylinders and can therefore fill the middle and center gaps, as shown in Fig. 20 As shown (for ease of illustration, a 2D slice of the results is shown).

[0261] Therefore, more generally, it should be appreciated that the flood-fill algorithm described herein employs a hierarchical spatial partitioning data structure, typically a quadtree for 2D images or an octree for stereo images, to decompose the flood-fill problem into work units that can be processed in parallel (e.g., by a GPU), with elevation depth data generated in response to occupancy of features in the image at different levels of the data structure, enabling it to efficiently navigate and effectively identify blank image areas.

[0262] As to when and how the algorithm is used for 3D printing purposes, further variations can be considered.

[0263] For example, refer to Fig. 8A and 8BIn the voxel capture of the samurai, the character's axe may be a separate object that is actually slightly spaced apart from the character within the video game environment (e.g., because the character is modeled to grip many different items and not all items will fit perfectly, or because of the specific position of the character / axe for printing at the moment of capture, or because the character has just started throwing the axe, etc.).

[0264] However, because of this, when the samurai is printed, the axe will fall out of the samurai's hand (or at least be loose).

[0265] This issue can be problematic for common elements of video game characters that are assembled or juxtaposed in the game world rather than originating from a single model; one example is that the eyes of a video game character are often separate from the character's head; depending on the capture process, any gap between the face and the eye can cause the eye to be printed loosely and then fall into the hollow body of the character.

[0266] Therefore, more generally, referring now to Fig.21 , in example (a), a 2D piece of the model described above for explanation purposes, a character or other printable object may be a composite of several directly adjacent elements A and D, but the user may also want them to have an associated attachment C (or another element that is also not fully aligned with the character). At the same time, another element B of the environment that may be located within the overall area of ​​the character / object may be considered separate, or may be unwanted or considered a secondary printable object (e.g., a football that is about to be kicked by the character).

[0267] Running the flood-fill algorithm as described herein results in the expected identification of three objects as shown in example (b): the complex of A and D, and C and B separately.

[0268] Recall that the flood fill algorithm can be initialized at specific leaves / points in the image (rather than from the border), and then the algorithm can be used for separate purposes:

[0269] After the closed areas are determined by the first flood fill, a second flood fill can be performed within one of the closed areas (e.g., the largest closed area corresponding to A&D) using a different color or value as the ID. The exact starting point within the area is not important, although a test is optionally performed to ensure that it is greater than N pixels away from any existing source fill target color to ensure that the first pixel is not an anti-aliased alias of the color, but is actually a representation of the area. Obviously, in the case of the volume model described in this article, such anti-aliasing is not a problem, and therefore this consideration is not necessary.

[0270] Since parts B and C are not connected, they will not be filled, as Fig.21. Pixels that have not yet been filled can then be identified and the algorithm applied again with a different color / ID until no pixels are left that are flood-filled with the source color / ID and therefore all disconnected segments have been identified. As a result, each of segments B and C will receive its own fill color / ID, with the final result being Fig.21 At this point, the pixels / voxels occupied by each ID can also be counted (or the counts already made during the statistical filling process) to determine which is the main part (if relevant to any further processing).

[0271] It should be appreciated that the above approach also extends to octree and stereo image versions of the algorithm.

[0272] Given the complete characterization of a patch of an image created in this way, it is then possible to determine where separated objects are within a threshold distance of each other, and apply rules to determine whether these objects should actually be connected.

[0273] For example, a metric based on the distance between two objects divided by the (optionally weighted) length or area of ​​the objects within that distance or a predetermined threshold maximum distance may be compared to a threshold to determine whether two objects may be connected.

[0274] In other words, when two irregularly shaped objects happen to be close to each other at a single point, the distance between them is offset by dividing the short shared length within that distance, so that in reality, the objects must be closer to each other to trigger their combination. This also has a negative impact on connecting objects through narrow and therefore potentially fragile links that are fragile. In contrast, for example, if two objects are a medium distance apart, but parallel to each other at a significant length (usually indicating an intention to treat the objects as one), then the medium distance divided by that length makes it more likely that the objects will be connected.

[0275] Other ways will be obvious to the skilled person, such as for example simply setting a threshold distance between two objects within which they should be merged.

[0276] In any case, it should be understood that, optionally, supplemental geometries of the kind previously described herein may then be used to provide the connection and / or thicken and strengthen the connection.

[0277] Alternatively or additionally, the smaller of the two objects may be moved to merge with the larger object, and the amount of movement may optionally be the amount of movement required to merge the objects until the cross-sectional area of ​​the small object at the intersection reaches a threshold size deemed to provide a sufficiently strong connection. Optionally, the smaller object may be subjected to an enlargement or rescaling function to increase its cross-sectional area in the vicinity of the larger object. This may directly result in the smaller object being merged with the larger object, or simply increase the cross-sectional area of ​​the small object, thereby reducing the movement required to achieve a strong connection.

[0278] Similarly, the smaller of the two objects may be extruded to intersect the larger object, and also optionally, the extrusion may begin at a point in the smaller object having a predetermined cross-sectional area that is deemed to provide a sufficiently strong connection.

[0279] While the above has been described with respect to adjusting features of a small object, it should be appreciated that features of a larger object proximate to a smaller object may alternatively or additionally be adjusted to connect the two objects.

[0280] exist Fig.21 The enlargement or rescaling option is shown in examples (e) and (f), in which case the result is that the entire object C becomes larger and intersects the host object with a large enough cross-sectional area to form a strong connection.

[0281] Optionally in this regard, in the event that moving or resizing an object causes other changes in connectivity throughout the object, the process of identifying the object within the image may be rerun using a flood fill algorithm.

[0282] The result is a single body that combines the original objects A, C, and D (as shown in example (g)). Meanwhile, the more distant object B does not satisfy the distance / shared length metric, or the absolute distance metric, or any other suitable metric for merging, and therefore remains separate, as shown in Fig.21 As shown in example (h).

[0283] It should be understood that this approach is preferably implemented using the octree and stereo versions of the algorithm, because in practice, the different elements of a real video game character are unlikely to be flat and parallel to each other, and therefore two elements in such a model may have different distances from each other in consecutive cross-sections. Therefore, moving, scaling or squeezing a single slice according to the conditions specific to that slice may lead to inconsistent results in the entire model. Therefore, although the constraints may be inherited between consecutive slices, it is much simpler to implement this process in stereo.

[0284] It should also be understood that the model can be presented to the user for manual editing, and the above reference can be used Fig.21The fill algorithm of instance (d) is used to mark to identify any / each separate enclosed area that may require attention, and the resulting changes in the model caused by the user's actions are re-evaluated in the same manner.

[0285] Now also refer to Fig. 22 In an overview embodiment of the present invention, a flood fill method for a parallel implementation includes:

[0286] In a first step s2210, an image is received (e.g., a 2D slice or a 3D stereoscopic image of an object in a game, optionally before, during or after other modifications have been applied as described elsewhere herein), the image comprising first elements having a floodfill source color to be replaced with a floodfill target color, and second elements not having the floodfill source color (e.g., boundary pixels as described previously);

[0287] In a second step s2220, the image is partitioned according to a hierarchical spatial partitioning scheme (e.g., a quadtree or an octree) to form units at multiple levels in the hierarchy (such as Fig.16 those shown in );

[0288] In a third step s2230, the occupation of each cell by the second element is detected (e.g., using occupied_all_bits and occupied_some_bits as described previously herein);

[0289] For any cell that is not occupied by a second element, in a fourth step s2240, the depth-elevation value of the cell's child cells is set to a depth-elevation value greater than the cell's value, indicating that the floodfill will navigate the hierarchy toward the root cell by the cell's depth-elevation amount while occupying only nodes that do not have any second elements, thereby reaching the largest parent cell that can be floodfilled (e.g., as previously described herein and as Fig.16 shown); and

[0290] In a fifth step s2250, the cells not occupied by the second element are filled with the floodfill source color with the floodfill target color, thereby modifying the received image.

[0291] In this example of an outlined embodiment, the image is two-dimensional, the hierarchical space partitioning scheme is a quadtree, and the cells are constructed in lines.

[0292] In the example of this outlined embodiment, the image is stereoscopic, the hierarchical space partitioning scheme is an octree, and the cells are cuboids.

[0293] In this example of the outlined embodiment, when a cell is processed, at least one side from which the cell was entered is stored, and additional cells for processing are identified as those neighboring cells other than the cells on the side from which the cell was entered.

[0294] In an example of this outlined embodiment, as a cell is processed, storage is entered from at least one side of the cell, and work items are generated for sub-cells adjacent to at least one side of the storage.

[0295] In the example of the outlined embodiment, a work item is generated for each of the plurality of units, and two or more corresponding work items are processed in parallel.

[0296] In this case, optionally, two or more corresponding work items are processed by the graphics processing unit.

[0297] In the example of this outlined embodiment, the image elements replaced with the flood fill target color represent the flood fill mask.

[0298] In this version of the example, image regions outside the flood fill mask are defined as closed regions.

[0299] In this version, optionally, a distance-based metric is determined for the distance between the two enclosed regions, and if the metric is less than a predetermined threshold amount, at least one of the two regions is modified by one or more modifications in a list, including: moving at least one of the two regions until the two regions intersect; and increasing the size of at least a portion of at least one of the two regions until the two regions intersect.

[0300] Optionally, the flood-fill algorithm is repeated if at least one of the two regions is modified.

[0301] In this version, the image can represent part or all of the object to be 3D printed.

[0302] Optionally, the image is then modified by adding holes formed by one or more elements having the flood fill target color to areas of the image identified as closed regions. This can be used to create a drainage pattern for 3D printed material that might otherwise be trapped within the closed region (see the "Waste" section later in this document). Optionally, adding a hole can be conditional on the closed region being greater than a threshold size or volume. The hole can be a fixed length (e.g., equal to or longer than a known shell thickness), or can be extended by replacing a predetermined width of pixels (e.g., boundary pixels) with the flood fill target color until a width of pixels is detected that includes the flood fill source color, indicating that the "wall" of the closed region has been broken.

[0303] If the image represents some or all of an object to be 3D printed, then optionally the method may include the steps of generating 3D printing instructions in response to the flood fill image; and outputting those 3D printing instructions to a 3D printer.

[0304] Object Selection

[0305] The above discussion assumes that target object(s) have been selected from an area of ​​the virtual environment; for example, a user may choose to print only their avatar, or their avatar and an opponent.

[0306] For example, a bounding box may be defined around the desired object(s) through a user interface, or a cursor or guideline may be used through a user interface to indicate a particular object in the game environment for selection.

[0307] If (multiple) objects isolated from a supporting surface are selected in this manner, a default surface such as a plane can be generated having an area corresponding to the size of the selected (multiple) objects. The selected (multiple) objects can then be coupled to the supporting surface using the techniques described herein.

[0308] edit

[0309] Once the printer geometry is obtained using the above techniques (if applicable), it can optionally be transferred to an editing application so that the user can preview or modify the model before sending it to the printer. For example, it may be necessary to check the model for possible errors, and / or edit the poses and positions of certain key characters. The editor can also add labels only to the painted texture, or to the surface structure of the object, such as by deleting surface voxels, so that the user can effectively carve information into the surface of the model.

[0310] Final Print

[0311] Once the user is satisfied with their 3D model, it can be sent to the printer. As mentioned earlier, the printer driver analyzes the model and divides it into voxel layers for continuous printing.

[0312] If the printer is a home device physically attached to the console or a shared home network, a printer driver may be implemented on the entertainment device and the entertainment device sends the generated draw list to the printer.

[0313] However, alternatively, printing services may also be provided by the network of the entertainment device (that is, for example, a network provided by the manufacturer of the entertainment device). This may allow access to more expensive and complex 3D printers than the average consumer can afford, and therefore allow higher quality models. In this case, the printer driver may still be on the console to distribute the processing load between users, or it may be located on a central server connected to the 3D printer. In the first case, the local printer driver will generate a drawing list that can be securely sent to the central print queue server together with metadata related to the user's mailing address. In the second case, the entertainment device securely sends the 3D model printer geometry to the central printer driver, which performs relevant analysis to generate these lists before queuing the required drawing lists for printing. In either of these two cases, the printing of the model may depend on the payment of the fee, such as via a payment card registered on the network of the entertainment device, or similarly may depend on the receipt of a voucher, which is won, for example, as a reward in a trophy or other game, or obtained as part of the purchase price of the game, thereby authorizing the user to create a predetermined number of 3D models from the game.

[0314] Therefore, the step of 3D printing a model based on the output 3D printable model data directly produces a 3D printed model.

[0315] scrap

[0316] It should be understood that 3D printing techniques such as selective laser sintering can trap large amounts of raw printer material within the resulting model. This is because during the printing process, these techniques apply successive layers of powder over the top of the model, fusing only small portions corresponding to the periphery of the model, while leaving the rest of the powder inside the inaccessible periphery. The overall result is a large volume of powder, inside which is a fused shell, the powder outside of which can be removed, while the powder inside the shell is trapped.

[0317] This creates unnecessary costs in terms of wasted raw materials. It can also affect the balance of the resulting object. Therefore, in models designed for 3D printing, vents or openings are often included to allow for the dumping or flushing of spent powder.

[0318] However, such vents or openings do not exist in video game characters and are undesirable.

[0319] Therefore, in an embodiment of the present invention, if a powder-based printer is to be used, a programmatic rule is implemented to calculate the effective volume of space within an object in the printed scene. When this volume exceeds a threshold amount, the object is considered a hollow body, and the programmatic rule creates an opening in the voxel shell of the hollow body of a size that allows the printer powder to flow out. Preferably, the opening is located on the bottom surface of the hollow body and / or on the side facing away from the original viewpoint displayed on the screen when the user indicates that they want to print the scene. The opening can be a hole in the voxel shell, or preferably an annular void forming a ring or other closed loop shape. In the case of an annular void, this will generate a loose cover in the printed body that can be removed to allow powder to drain out, but the cover can then be bonded back to the model, thereby minimizing the aesthetic impact of removing powder on the finished product.

[0320] It should be understood that a 3D printed scene may contain multiple such hollow bodies. For example, a scene may include a ground surface, a character (such as Blobman), and a tree. The ground surface is typically an open shell. Meanwhile, the tree is placed on it, with the ground surface forming the base of the tree. By calculating the volume in the model, the trunk of the tree can be identified as a hollow object. Thus, a hole can be created in the ground under the tree, which is only visible when the model is viewed from below. In contrast, the Blobman character is supported by connecting rods on the ground. Therefore, if the character trunk is identified as a hollow body, an annular void is included in the direction away from the original viewpoint, optionally at a threshold distance from any connecting rod. This allows the trunk to expel powder, but it can also be repaired by re-fixing the removable components of the torso formed by the annular void into place.

[0321] Alternatively, procedural rules may use features of the object itself to reduce the aesthetic impact of the hole or annular void, for example, by setting part or all of one or more boundaries of the hole or void to follow adjacent boundaries between texture colors in the model, and / or to follow adjacent ridges or other discontinuities in the model surface.

[0322] The system can use the 3D model data to estimate the amount of printer raw materials needed and provide a quote for the estimated cost of printing the model before the user commits to doing so.

[0323] Finally, refer back Figure 1 It should be understood that an apparatus or system of apparatuses may implement the outlined embodiments or variant instances, versions and options of the above-described methods, and more generally the flood-fill techniques described herein.

[0324] Thus, for example, in another overview embodiment, an entertainment device (such as a PlayStation )10 includes: a flood fill processor (such as GPU 20B operating under appropriate software instructions), which is adapted in operation to: receive an image, the image including a first element having a flood fill source color to be replaced with a flood fill target color, and a second element not having the flood fill source color; divide the image according to a hierarchical spatial partitioning scheme to form cells at multiple levels in the hierarchy; detect the occupancy of each cell by the second element, and for any cell not occupied by the second element, set the depth elevation value of the cell's child cells to a depth elevation value greater than the value of the cell, indicating that the flood fill should navigate the hierarchy toward the root cell by the depth elevation amount while occupying only nodes without any second elements to reach the largest parent cell that can be flood filled; and fill the flood fill source color of cells not occupied by the second element with the flood fill target color, thereby modifying the received image.

[0325] Thus, a general purpose computer operating under appropriate software instructions can implement the flood fill method described above.

[0326] Other methods and techniques described herein may also be implemented by device 10 operating under appropriate software instructions, and 3D printing itself may be accomplished by a device operating in conjunction with 3D printer 100 .

[0327] It will be appreciated from the foregoing description that the methods described herein may be performed on conventional hardware adapted appropriately by software instructions or by the inclusion or replacement of dedicated hardware.

[0328] Thus, the required configuration of the existing portion of the conventional equivalent device may be implemented in the form of a computer program product comprising processor-implementable instructions stored on a non-transitory machine-readable medium (such as a floppy disk, optical disk, hard disk, PROM, RAM, flash memory, or any combination of these or other storage media), or implemented in hardware as an ASIC (application specific integrated circuit) or FPGA (field programmable gate array) or other configurable circuit suitable for configuring the conventional equivalent device. In addition, such computer programs may be transmitted via data signals on a network (such as an Ethernet, a wireless network, the Internet, or any combination of these or other networks).

Claims

1. A flood filling method for parallel implementation, comprising the steps of: receiving an image including a first element having a flood fill source color to be replaced with a flood fill destination color and a second element not having the flood fill source color; dividing the image into a hierarchy according to a hierarchical spatial partitioning scheme to form cells at a plurality of levels in the hierarchy, each level in the plurality of levels having cells of respective sizes from a smallest size at a lowest level in the hierarchy to a largest size at a highest level in the hierarchy; detecting occupancy of each cell by the second element; For any child cell of each parent cell not occupied by the second element, setting a corresponding depth-elevation value, the depth-elevation value of the child cell being greater than the depth-elevation value of the parent cell, indicating that the floodfill should navigate the hierarchy toward the root cell by the depth-elevation value while occupying only cells not occupied by the second element, thereby reaching the largest parent cell that is not occupied by the second element and is therefore available for floodfilling; as well as The received image is modified by filling the floodfill source color of the largest parent cell not occupied by the second element with the floodfill destination color.

2. The method of claim 1, wherein the image is two-dimensional, the hierarchical space partitioning scheme is a quadtree, and the cells are constructed of straight lines.

3. The method of claim 1, wherein the image is stereoscopic, the hierarchical space partitioning scheme is an octree, and the cells are cuboids.

4. The method according to claim 1, wherein: When processing cells, at least one side from which storage entered the given cell is stored, and additional cells for processing are identified as those neighboring cells other than cells on at least one side from which storage entered the given cell.

5. The method according to claim 1, wherein: As cells are processed, storage is entered from at least one side of a given cell, and work items are generated for subcells adjacent to at least one side of the storage. The method of claim 1 , wherein a work item is generated for each of the plurality of units, and two or more corresponding work items are processed in parallel. The method of claim 6 , wherein the two or more corresponding work items are processed by a graphics processing unit.

8. The method of claim 1, wherein the image elements replaced with the flood fill target color represent a flood fill mask.

9. The method of claim 8, wherein the image area outside the flood fill mask is defined as a closed area.

10. The method according to claim 9, wherein: Determining a distance-based metric for a distance between two enclosed areas, and if the metric is less than a predetermined threshold amount, modifying at least one of the two enclosed areas by one or more modifications in a list, comprising: i. moving at least one of the two closed areas until the two closed areas intersect; and ii. Increasing the size of at least a portion of at least one of the two enclosed areas until the two enclosed areas intersect.

11. The method according to claim 10, wherein: If at least one of the two closed regions is modified, the flood-fill algorithm is repeated.

12. The method of claim 9, wherein the image represents part or all of an object to be 3D printed.

13. The method according to claim 12, comprising the steps of: The image is modified by adding holes formed by one or more elements having the flood fill target color to areas of the image identified as closed areas.

14. The method according to claim 12, comprising the steps of: generating 3D printing instructions in response to the flood fill image; and Output those 3D printing instructions for the 3D printer.

15. A computer program product comprising program instructions, which, when executed by a computer, implement the method according to any one of claims 1 to 14.

16. An entertainment device comprising: A flood fill processor is operationally adapted to: receiving an image including a first element having a flood fill source color to be replaced with a flood fill destination color and a second element not having the flood fill source color; dividing the image into a hierarchy according to a hierarchical spatial partitioning scheme to form cells at a plurality of levels in the hierarchy, each level in the plurality of levels having cells of respective sizes from a smallest size at a lowest level in the hierarchy to a largest size at a highest level in the hierarchy; detecting occupancy of each cell by the second element; For any child cell of each parent cell not occupied by the second element, setting a corresponding depth-elevation value, the depth-elevation value of the child cell being greater than the depth-elevation value of the parent cell, indicating that the floodfill should navigate the hierarchy toward the root cell by the depth-elevation value while occupying only cells not occupied by the second element, thereby reaching the largest parent cell that is not occupied by the second element and is therefore available for floodfilling; as well as The received image is modified by filling the floodfill source color of the largest parent cell not occupied by the second element with the floodfill destination color.

Images