Method, device and equipment for generating image point cloud data
By reshaping and stitching the image data matrix and combining it with a large language model, image point cloud data is generated, which solves the problem of high-cost annotation in existing technologies and realizes the generation of unlabeled data.
Patent Information
- Application Number
- CN202511112106.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-08
- Publication Date
- 2025-11-21
AI Technical Summary
In existing technologies, the acquisition cost of image point cloud data is high, requiring a large amount of sample data for annotation, resulting in high annotation costs.
By reshaping and stitching an image data matrix based on a preset image, a zero-based data matrix is constructed and then input into a trained large language model to generate a point cloud data matrix of the image to be processed, without the need for sample data annotation.
It enables the generation of image point cloud data without the need for pre-labeled sample data, thus reducing labeling costs.
Smart Images
Figure CN120997579A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer data processing technology, and in particular to a technique for generating image point cloud data. Background Technology
[0002] With the continuous development of digitalization, point cloud data from images is needed in fields such as intelligent manufacturing, digital building, autonomous driving, and environmental perception. For example, in the field of autonomous driving / environmental perception, point cloud data from images can be used to annotate the 3D boundaries of targets such as vehicles and pedestrians in the environment, which can then be combined with dynamic tracking to achieve obstacle avoidance and path planning. In building digitalization, when constructing BIM (Building Information Modeling) models of buildings, point cloud data from images can be used to extract structural details to generate high-precision (up to millimeter level) models for construction monitoring, collision detection, and other applications.
[0003] Currently, acquiring point cloud data for images primarily relies on various sensor technologies and algorithmic models. This requires pre-acquiring a large amount of sample data, labeling it, and constructing a sample dataset to train the algorithmic model, resulting in significant labeling costs. Reducing or even eliminating the labeling costs of acquiring point cloud data for images remains a challenging technical problem. Summary of the Invention
[0004] The purpose of this invention is to provide a method, apparatus, and device for generating image point cloud data, so as to at least partially solve the technical problem of high cost of acquiring existing image point cloud data.
[0005] According to one aspect of the present invention, a method for generating image point cloud data is provided, wherein the method comprises:
[0006] A. Based on the image data matrix of a preset image, obtain the point cloud data matrix of the preset image, and perform a reshape operation on the image data matrix of the preset image to obtain a first data matrix with the same shape as the point cloud data matrix of the preset image. Then, perform a stitching operation on the first data matrix and the point cloud data matrix of the preset image to obtain a first stitched data matrix.
[0007] B acquires the image data matrix of the image to be processed and constructs an all-zero data matrix with the same shape as the point cloud data matrix of the preset image;
[0008] C performs a reshape operation on the image data matrix of the image to be processed to obtain a third data matrix with the same shape as the point cloud data matrix of the preset image, and performs a splicing operation on the third data matrix and the all-zero data matrix to obtain a second spliced data matrix;
[0009] After sequentially concatenating the first and second concatenated data matrices, the data is input into the trained large language model to obtain the model output. A data matrix with the same shape as the point cloud data matrix of the preset image is extracted from the model output and used as the point cloud data matrix of the image to be processed.
[0010] Optionally, the method for generating image point cloud data further includes:
[0011] In step A, based on the image data matrix of the preset image, the depth data matrix of the preset image is obtained and processed to obtain a second data matrix with the same shape as the point cloud data matrix of the preset image;
[0012] In step C, based on the image data matrix of the image to be processed, the depth data matrix of the image to be processed is obtained and processed to obtain a fourth data matrix with the same shape as the point cloud data matrix of the preset image;
[0013] In step A, the step of concatenating the first data matrix and the point cloud data matrix of the preset image to obtain the first concatenated data matrix includes:
[0014] The first data matrix, the second data matrix, and the point cloud data matrix of the preset image are stitched together to obtain the first stitched data matrix;
[0015] In step C, the concatenation operation between the third data matrix and the all-zero data matrix to obtain the second concatenated data matrix includes:
[0016] The third data matrix, the fourth data matrix, and the all-zero data matrix are concatenated to obtain the second concatenated data matrix.
[0017] According to another aspect of the present invention, an apparatus for generating image point cloud data is provided, wherein the apparatus comprises:
[0018] The first module is used to obtain the point cloud data matrix of the preset image based on the image data matrix of the preset image, and to perform a reshape operation on the image data matrix of the preset image to obtain a first data matrix with the same shape as the point cloud data matrix of the preset image, and to perform a splicing operation on the first data matrix and the point cloud data matrix of the preset image to obtain a first spliced data matrix.
[0019] The second module is used to acquire the image data matrix of the image to be processed and construct an all-zero data matrix with the same shape as the point cloud data matrix of the preset image;
[0020] The third module is used to perform a reshape operation on the image data matrix of the image to be processed to obtain a third data matrix with the same shape as the point cloud data matrix of the preset image, and to perform a splicing operation on the third data matrix and the all-zero data matrix to obtain a second spliced data matrix.
[0021] The fourth module is used to sequentially concatenate the first concatenated data matrix and the second concatenated data matrix and input them into the trained large language model to obtain the model output. The module then extracts a data matrix with the same shape as the point cloud data matrix of the preset image from the model output and uses it as the point cloud data matrix of the image to be processed.
[0022] Optionally, the first module is further configured to:
[0023] Based on the image data matrix of a preset image, the depth data matrix of the preset image is obtained and processed to obtain a second data matrix with the same shape as the point cloud data matrix of the preset image. The step of concatenating the first data matrix and the point cloud data matrix of the preset image to obtain a first concatenated data matrix includes:
[0024] The first data matrix, the second data matrix, and the point cloud data matrix of the preset image are stitched together to obtain the first stitched data matrix;
[0025] The third module is also used for:
[0026] Based on the image data matrix of the image to be processed, the depth data matrix of the image to be processed is obtained and processed to obtain a fourth data matrix with the same shape as the point cloud data matrix of the preset image. The step of concatenating the third data matrix and the all-zero data matrix to obtain a second concatenated data matrix includes:
[0027] The third data matrix, the fourth data matrix, and the all-zero data matrix are concatenated to obtain the second concatenated data matrix.
[0028] Compared with the prior art, the present invention provides a method, apparatus, and device for generating image point cloud data. The method includes: A) obtaining a point cloud data matrix of a preset image based on an image data matrix, reshaping the image data matrix of the preset image to obtain a first data matrix with the same shape as the point cloud data matrix of the preset image, and concatenating the first data matrix and the point cloud data matrix of the preset image to obtain a first concatenated data matrix; B) obtaining an image data matrix of an image to be processed, and constructing a zero-based data matrix with the same shape as the point cloud data matrix of the preset image; C) reshaping the image data matrix of the image to be processed to obtain a third data matrix with the same shape as the point cloud data matrix of the preset image, and concatenating the third data matrix and the zero-based data matrix to obtain a second concatenated data matrix; D) sequentially concatenating the first concatenated data matrix and the second concatenated data matrix and inputting them into a trained large language model to obtain model output, and extracting a data matrix with the same shape as the point cloud data matrix of the preset image from the model output as the point cloud data matrix of the image to be processed. This invention uses a large language model for modeling, which can obtain the point cloud data of the image without labeling the sample data, thus eliminating the labeling cost. Attached Figure Description
[0029] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:
[0030] Figure 1 A schematic flowchart of a method for generating image point cloud data according to one aspect of the present invention is shown.
[0031] Figure 2 This diagram illustrates an exemplary data matrix splicing method according to one aspect of the present invention.
[0032] Figure 3 A schematic diagram of an apparatus for generating image point cloud data according to another aspect of the present invention is shown.
[0033] The same or similar reference numerals in the accompanying drawings represent the same or similar parts. Detailed Implementation
[0034] The present invention will now be described in further detail with reference to the accompanying drawings.
[0035] In a typical configuration of various embodiments of the present invention, the method execution entity, each trusted party of the system and / or each module of the device all include one or more processors (CPU), input / output interfaces, network interfaces and memory.
[0036] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.
[0037] Computer-readable media include both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include non-transitory computer-readable media, such as modulated data signals and carrier waves.
[0038] To further illustrate the technical means adopted and the effects achieved by the present invention, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings and various embodiments.
[0039] Figure 1 The diagram illustrates a method for generating image point cloud data according to one aspect of the present invention, wherein one embodiment of the method includes:
[0040] S101 Based on the image data matrix of the preset image, obtain the point cloud data matrix of the preset image, and perform a reshape operation on the image data matrix of the preset image to obtain a first data matrix with the same shape as the point cloud data matrix of the preset image, and perform a stitching operation on the first data matrix and the point cloud data matrix of the preset image to obtain a first stitched data matrix.
[0041] S102 Obtains the image data matrix of the image to be processed, and constructs an all-zero data matrix with the same shape as the point cloud data matrix of the preset image;
[0042] S103 performs a reshape operation on the image data matrix of the image to be processed to obtain a third data matrix with the same shape as the point cloud data matrix of the preset image, and performs a splicing operation on the third data matrix and the all-zero data matrix to obtain a second spliced data matrix.
[0043] S104 sequentially concatenates the first concatenated data matrix and the second concatenated data matrix, then inputs them into the trained large language model to obtain the model output. From the model output, a data matrix with the same shape as the point cloud data matrix of the preset image is extracted as the point cloud data matrix of the image to be processed.
[0044] In this application, the various method embodiments and / or optional embodiments are executed through device 100, which is a computer device and / or cloud equipped with a corresponding hardware and software environment for processing image data. The computer device includes, but is not limited to, personal computers, laptops, industrial computers, network hosts, single network servers, and sets of multiple network servers. The cloud consists of a large number of computers or network servers based on cloud computing, where cloud computing is a type of distributed computing, consisting of a virtual supercomputer composed of a group of loosely coupled computer sets. Here, the computer device and / or cloud are merely examples; other existing or future devices and / or resource sharing platforms applicable to this application should also be included within the scope of protection of this application, and are hereby incorporated by reference.
[0045] In this embodiment, in step S101, the device 100 obtains the point cloud data matrix of the preset image based on the image data matrix of the preset image, performs a reshape operation on the image data matrix of the preset image to obtain a first data matrix with the same shape as the point cloud data matrix of the preset image, and performs a stitching operation on the first data matrix and the point cloud data matrix of the preset image to obtain a first stitched data matrix.
[0046] Specifically, a preset image as a sample can be pre-stored in the device 100, or the device 100 can pre-obtain the preset image from a storage device or storage system. The image data of the preset image can be represented in a data matrix format. First, the device 100 can obtain the point cloud data of the preset image based on the image data of the preset image. The point cloud data of the preset image can also be represented in a data matrix format. Next, the device 100 can perform a reshape operation on the image data matrix of the preset image to obtain a first data matrix with the same shape as the point cloud data matrix of the preset image, which can ensure subsequent matrix stitching operations. Then, the device 100 can perform a stitching operation on the first data matrix and the point cloud data matrix of the preset image to obtain a first stitched data matrix.
[0047] For example, image data can be represented using a three-dimensional data matrix (H, W, 3), where H and W represent the pixel dimensions of the image, and 3 represents the RGB components of a pixel. An element in the image data matrix represents the pixel value of the R, G, or B component of a pixel. Point cloud data of the image can be calculated using OpenCV tools based on the image data, and can be represented using a two-dimensional data matrix (N, 3), where N = H * W, and 3 represents the XYZ spatial coordinates. Figure 2 As shown, the image data matrix (H, W, 3) of a preset image can be reshaped. For example, using the `reshape` method in Python's NumPy library, the three-dimensional image data matrix (H, W, 3) of the preset image can be transformed into a two-dimensional first data matrix (H*W, 3), which has the same shape as the point cloud data matrix of the preset image. The first data matrix can be concatenated with the point cloud data matrix of the image according to a certain dimensional order. For example, keeping the "row" dimension unchanged, the first data matrix and the point cloud data matrix of the preset image can be concatenated according to the "column" dimension to obtain a first concatenated data matrix with the shape (H*W+N, 3).
[0048] Continuing with this embodiment, in step S102, device 100 acquires the image data matrix of the image to be processed and constructs an all-zero data matrix with the same shape as the point cloud data matrix of the preset image.
[0049] The device 100 can acquire the image data matrix of the image to be processed and construct an all-zero data matrix, wherein the shape of the all-zero data matrix should be the same as the shape of the point cloud data matrix of the aforementioned preset image.
[0050] Continuing with the above example, the image data of the image to be processed can be represented by a three-dimensional data matrix (H, W, 3), and the all-zero data matrix can be represented by a two-dimensional data matrix (N, 3).
[0051] Continuing with this embodiment, in step S103, the device 100 performs a reshape operation on the image data matrix of the image to be processed to obtain a third data matrix with the same shape as the point cloud data matrix of the preset image, and performs a splicing operation on the third data matrix and the all-zero data matrix to obtain a second spliced data matrix.
[0052] Specifically, device 100 can perform a reshape operation on the image data matrix of the image to be processed to obtain a third data matrix with the same shape as the point cloud data matrix of the preset image, to ensure subsequent matrix stitching operations. Then, the third data matrix and the all-zero data matrix are stitched together to obtain a second stitched data matrix.
[0053] Continuing with the above example, the image data matrix (H, W, 3) of the image to be processed can be reshaped to transform the three-dimensional image data matrix (H, W, 3) into a two-dimensional third data matrix (H*W, 3), which has the same shape as the point cloud data matrix of the preset image. The third data matrix can be concatenated with the all-zero data matrix in the same dimensional order. For example, keeping the "row" dimension unchanged, the third data matrix and the all-zero data matrix are concatenated in the "column" dimension order to obtain a second concatenated data matrix, which also has the shape (H*W+N, 3).
[0054] Continuing with this embodiment, in step S104, the device 100 sequentially splices the first spliced data matrix and the second spliced data matrix and then inputs them into the trained large language model to obtain the model output. In the model output, a data matrix with the same shape as the point cloud data matrix of the preset image is extracted as the point cloud data matrix of the image to be processed.
[0055] The device 100 can sequentially stitch together the first and second stitched data matrices, and then input the stitched data matrix into a large language model that has undergone targeted pre-training. The model outputs a data matrix with the same shape as the stitched data matrix. Then, referring to the stitching order, the device extracts data at the corresponding positions from the data matrix output by the large language model to obtain a data matrix with the same shape as the point cloud data matrix of the preset image. This data matrix is used as the point cloud data matrix of the image to be processed to obtain the point cloud data of the image to be processed.
[0056] Continuing with the above example, the first concatenated data matrix with shape (H*W+N, 3) and the second concatenated data matrix with shape (H*W+N, 3) can be sequentially concatenated. For example, keeping the "row" dimension unchanged, the first and second concatenated data matrices can be concatenated sequentially according to the "column" dimension to obtain a concatenated data matrix with shape (H*W+N+H*W+N, 3). This concatenated data matrix is then input into a large language model that has undergone targeted pre-training, such as Google's T5, Alibaba's Tongyi Qianwen, or Baidu's Wenxin Yiyan. The data matrix output by the large language model also has shape (H*W+N+H*W+N, 3). Then, the last N*3 elements of the data matrix output by the large language model are extracted to obtain a data matrix with shape (N, 3). This data matrix can be used as the point cloud data matrix of the image to be processed to obtain the point cloud data of the image to be processed.
[0057] Optionally, the method for generating image point cloud data further includes:
[0058] In step S101, based on the image data matrix of the preset image, the depth data matrix of the preset image is obtained and processed to obtain a second data matrix with the same shape as the point cloud data matrix of the preset image;
[0059] In step S103, based on the image data matrix of the image to be processed, the depth data matrix of the image to be processed is obtained and processed to obtain a fourth data matrix with the same shape as the point cloud data matrix of the preset image;
[0060] In step S101, the step of concatenating the first data matrix and the point cloud data matrix of the preset image to obtain the first concatenated data matrix includes:
[0061] The first data matrix, the second data matrix, and the point cloud data matrix of the preset image are stitched together to obtain the first stitched data matrix;
[0062] In step S103, the concatenation operation between the third data matrix and the all-zero data matrix to obtain the second concatenated data matrix includes:
[0063] The third data matrix, the fourth data matrix, and the all-zero data matrix are concatenated to obtain the second concatenated data matrix.
[0064] In this optional embodiment, in step S101, the device 100 may further obtain the depth data matrix of the preset image based on the image data matrix of the preset image, and process the depth data matrix of the preset image to obtain a second data matrix with the same shape as the point cloud data matrix of the preset image. Then, the obtained first data matrix, second data matrix and the point cloud data matrix of the preset image are stitched together to obtain a first stitched data matrix. In step S103, the device 100 may further obtain the depth data matrix of the image to be processed based on the image data matrix of the image to be processed, and process the depth data matrix of the image to be processed to obtain a fourth data matrix with the same shape as the point cloud data matrix of the preset image. Then, the obtained third data matrix, fourth data matrix and the all-zero data matrix are stitched together to obtain a second stitched data matrix. This can improve the accuracy of the point cloud data of the image to be processed obtained in step S104.
[0065] Continuing with the above example, the depth data and point cloud data of an image can be calculated using OpenCV based on the image data. The depth data can be represented by a three-dimensional data matrix (H, W, 1), where 1 represents depth, and the point cloud data by a two-dimensional data matrix (N, 3). The depth data matrix (H, W, 1) and the image data matrix (H, W, 3) of a preset image can be reshaped. For example, using the `reshape` method in Python's NumPy library, the three-dimensional depth data matrix (H, W, 1) of the preset image can be reshaped to obtain a two-dimensional data matrix (H*W, 1). Three such two-dimensional data matrices can be concatenated to form a second data matrix of shape (H*W, 3). Similarly, using the `reshape` method in Python's NumPy library, the three-dimensional image data matrix (H, W, 3) of the preset image can be transformed into a two-dimensional first data matrix (H*W, 3), with the same shape as the point cloud data matrix of the preset image. The first data matrix, the second data matrix, and the point cloud data matrix of the preset image can be concatenated in a certain dimensional order. For example, keeping the "row" dimension unchanged, the first data matrix, the second data matrix, and the point cloud data matrix of the preset image can be concatenated in the "column" dimension order to obtain the first concatenated data matrix, which has the shape (H*W+H*W+N, 3). The image data matrix (H, W, 3) and the depth data matrix (H, W, 1) of the image to be processed can be processed in the same way as the preset image to obtain the third data matrix and the fourth data matrix, which have the same shape as the point cloud data matrix of the preset image. The third data matrix, the fourth data matrix, and the all-zero data matrix can be concatenated in the same dimensional order. For example, keeping the "row" dimension unchanged, the third data matrix and the all-zero data matrix can be concatenated in the "column" dimension order to obtain the second concatenated data matrix, which also has the shape (H*W+H*W+N, 3).Then, the first concatenated data matrix with shape (H*W+H*W+N, 3) and the second concatenated data matrix with shape (H*W+H*W+N, 3) can be concatenated sequentially. For example, keeping the "row" dimension unchanged, the first and second concatenated data matrices can be concatenated sequentially according to the "column" dimension to obtain a concatenated data matrix with shape (H*W+H*W+N+H*W+H*W+N, 3). This concatenated data matrix is then input into a large language model that has undergone targeted pre-training, such as Google's T5, Alibaba's Tongyi Qianwen, or Baidu's Wenxin Yiyan. The data matrix output by the large language model also has shape (H*W+H*W+N+H*W+H*W+N, 3). Then, the last N*3 elements of the data matrix output by the large language model are extracted to obtain a data matrix with shape (N, 3). This data matrix can be used as the point cloud data matrix of the image to be processed to obtain more accurate point cloud data of the image to be processed.
[0066] The method of this embodiment and / or optional embodiments can obtain point cloud data of images without the need for prior annotation of a large number of sample images.
[0067] Figure 3 An apparatus for generating image point cloud data according to another aspect of the present invention is shown, wherein the apparatus of one embodiment includes:
[0068] The first module 310 is used to obtain the point cloud data matrix of the preset image based on the image data matrix of the preset image, and to perform a reshape operation on the image data matrix of the preset image to obtain a first data matrix with the same shape as the point cloud data matrix of the preset image, and to perform a splicing operation on the first data matrix and the point cloud data matrix of the preset image to obtain a first spliced data matrix.
[0069] The second module 320 is used to acquire the image data matrix of the image to be processed and construct an all-zero data matrix with the same shape as the point cloud data matrix of the preset image;
[0070] The third module 330 is used to perform a reshape operation on the image data matrix of the image to be processed to obtain a third data matrix with the same shape as the point cloud data matrix of the preset image, and to perform a splicing operation on the third data matrix and the all-zero data matrix to obtain a second spliced data matrix.
[0071] The fourth module 340 is used to sequentially concatenate the first concatenated data matrix and the second concatenated data matrix and input them into the trained large language model to obtain the model output. The module then extracts a data matrix with the same shape as the point cloud data matrix of the preset image from the model output and uses it as the point cloud data matrix of the image to be processed.
[0072] The device in this embodiment is integrated into the aforementioned device 100. The device 100 pre-stores a preset image as a sample, or the device 100 pre-obtains the preset image from a storage device or storage system. The image data of the preset image can be represented in the form of a data matrix.
[0073] In this embodiment, through the first module 310 of the device, firstly, point cloud data of the preset image can be obtained based on the image data of the preset image. The point cloud data of the preset image can also be represented in the form of a data matrix. Next, a reshape operation can be performed on the image data matrix of the preset image to obtain a first data matrix with the same shape as the point cloud data matrix of the preset image, which can ensure the subsequent matrix stitching operation. Then, the first data matrix and the point cloud data matrix of the preset image can be stitched together to obtain a first stitched data matrix.
[0074] Continuing with this embodiment, the second module 320 of the device can acquire the image data matrix of the image to be processed and construct an all-zero data matrix, wherein the shape of the all-zero data matrix should be the same as the shape of the point cloud data matrix of the aforementioned preset image.
[0075] Continuing with this embodiment, the third module 330 of the device can perform a reshape operation on the image data matrix of the image to be processed to obtain a third data matrix with the same shape as the point cloud data matrix of the preset image, ensuring subsequent matrix stitching operations. Then, the third data matrix and the all-zero data matrix are stitched together to obtain a second stitched data matrix.
[0076] Continuing with this embodiment, the first and second spliced data matrices can be sequentially spliced together using the fourth module 340 of the device. The spliced data matrix is then input into a large language model that has undergone targeted pre-training. The model outputs a data matrix with the same shape as the spliced data matrix. Then, referring to the splicing order, data at the corresponding positions are extracted from the data matrix output by the large language model to obtain a data matrix with the same shape as the point cloud data matrix of the preset image. This data matrix is used as the point cloud data matrix of the image to be processed to obtain the point cloud data of the image to be processed.
[0077] Optionally, the first module 310 is further configured to:
[0078] Based on the image data matrix of a preset image, the depth data matrix of the preset image is obtained and processed to obtain a second data matrix with the same shape as the point cloud data matrix of the preset image. The step of concatenating the first data matrix and the point cloud data matrix of the preset image to obtain a first concatenated data matrix includes:
[0079] The first data matrix, the second data matrix, and the point cloud data matrix of the preset image are stitched together to obtain the first stitched data matrix;
[0080] The third module 330 is also used for:
[0081] Based on the image data matrix of the image to be processed, the depth data matrix of the image to be processed is obtained and processed to obtain a fourth data matrix with the same shape as the point cloud data matrix of the preset image. The step of concatenating the third data matrix and the all-zero data matrix to obtain a second concatenated data matrix includes:
[0082] The third data matrix, the fourth data matrix, and the all-zero data matrix are concatenated to obtain the second concatenated data matrix.
[0083] In this optional embodiment, the first module 310 can also obtain the depth data matrix of the preset image based on the image data matrix of the preset image, and process the depth data matrix of the preset image to obtain a second data matrix with the same shape as the point cloud data matrix of the preset image. Then, the obtained first data matrix, second data matrix and the point cloud data matrix of the preset image are stitched together to obtain a first stitched data matrix. The third module 330 can also obtain the depth data matrix of the image to be processed based on the image data matrix of the image to be processed, and process the depth data matrix of the image to be processed to obtain a fourth data matrix with the same shape as the point cloud data matrix of the preset image. Then, the obtained third data matrix, fourth data matrix and the all-zero data matrix are stitched together to obtain a second stitched data matrix. The fourth module can then obtain point cloud data of the image to be processed with higher accuracy.
[0084] In the various embodiments and / or optional embodiments of the above system, the parts of the method steps performed by each module that are not mentioned are the same as those in the aforementioned related method embodiments and / or optional embodiments, and will not be repeated here.
[0085] According to another aspect of the present invention, a computer-readable medium is also provided, the computer-readable medium storing computer-readable instructions that can be executed by a processor to implement some or all of the foregoing methods.
[0086] It should be noted that the various method embodiments and / or optional embodiments of the present invention can be implemented in part or in whole in software and / or a combination of software and hardware. The software program involved in the present invention can be executed by a processor to implement some or all of the steps or functions of the above embodiments and / or optional embodiments. Similarly, the software program of the present invention (including related data structures) can be stored in a computer-readable recording medium.
[0087] Furthermore, a portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide part or all of the methods and / or technical solutions according to the invention through the operation of the computer. The program instructions invoking the methods of the invention may be stored in a fixed or removable recording medium, and / or transmitted via a data stream in a broadcast or other signal-carrying medium, and / or stored in the working memory of a computer device operating according to the program instructions.
[0088] According to another aspect of the present invention, an apparatus for generating image point cloud data is also provided, wherein the apparatus includes: a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the apparatus is triggered to execute part or all of the methods and / or technical solutions as described in the foregoing embodiments and / or optional embodiments.
[0089] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments and / or optional embodiments described above, and that the invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description; thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims. Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in the apparatus claims may also be implemented by a single unit or device through software and / or hardware. The terms "first," "second," etc., are used to denote names and do not indicate any particular order.
Claims
1. A method for generating image point cloud data, characterized in that, The method includes: Based on the image data matrix of a preset image, obtain the point cloud data matrix of the preset image, and perform a reshape operation on the image data matrix of the preset image to obtain a first data matrix with the same shape as the point cloud data matrix of the preset image. Then, perform a stitching operation on the first data matrix and the point cloud data matrix of the preset image to obtain a first stitched data matrix. Obtain the image data matrix of the image to be processed, and construct an all-zero data matrix with the same shape as the point cloud data matrix of the preset image; The image data matrix of the image to be processed is reshaped to obtain a third data matrix with the same shape as the point cloud data matrix of the preset image. The third data matrix and the all-zero data matrix are then concatenated to obtain a second concatenated data matrix. The first and second spliced data matrices are sequentially concatenated and then input into the trained large language model to obtain the model output. A data matrix with the same shape as the point cloud data matrix of the preset image is extracted from the model output and used as the point cloud data matrix of the image to be processed.
2. The method according to claim 1, characterized in that, The method further includes: Based on the image data matrix of a preset image, the depth data matrix of the preset image is obtained and processed to obtain a second data matrix with the same shape as the point cloud data matrix of the preset image; Based on the image data matrix of the image to be processed, the depth data matrix of the image to be processed is obtained and processed to obtain a fourth data matrix with the same shape as the point cloud data matrix of the preset image; The step of concatenating the first data matrix and the point cloud data matrix of the preset image to obtain the first concatenated data matrix includes: The first data matrix, the second data matrix, and the point cloud data matrix of the preset image are stitched together to obtain the first stitched data matrix; The step of concatenating the third data matrix and the all-zero data matrix to obtain the second concatenated data matrix includes: The third data matrix, the fourth data matrix, and the all-zero data matrix are concatenated to obtain the second concatenated data matrix.
3. An apparatus for image and speech matching, characterized in that, The device includes: The first module is used to obtain the point cloud data matrix of the preset image based on the image data matrix of the preset image, and to perform a reshape operation on the image data matrix of the preset image to obtain a first data matrix with the same shape as the point cloud data matrix of the preset image, and to perform a splicing operation on the first data matrix and the point cloud data matrix of the preset image to obtain a first spliced data matrix. The second module is used to acquire the image data matrix of the image to be processed and construct an all-zero data matrix with the same shape as the point cloud data matrix of the preset image; The third module is used to perform a reshape operation on the image data matrix of the image to be processed to obtain a third data matrix with the same shape as the point cloud data matrix of the preset image, and to perform a splicing operation on the third data matrix and the all-zero data matrix to obtain a second spliced data matrix. The fourth module is used to sequentially concatenate the first concatenated data matrix and the second concatenated data matrix and input them into the trained large language model to obtain the model output. The module then extracts a data matrix with the same shape as the point cloud data matrix of the preset image from the model output and uses it as the point cloud data matrix of the image to be processed.
4. The apparatus according to claim 3, characterized in that, The first module is also used for: Based on the image data matrix of a preset image, the depth data matrix of the preset image is obtained and processed to obtain a second data matrix with the same shape as the point cloud data matrix of the preset image. The step of concatenating the first data matrix and the point cloud data matrix of the preset image to obtain a first concatenated data matrix includes: The first data matrix, the second data matrix, and the point cloud data matrix of the preset image are stitched together to obtain the first stitched data matrix; The third module is also used for: Based on the image data matrix of the image to be processed, the depth data matrix of the image to be processed is obtained and processed to obtain a fourth data matrix with the same shape as the point cloud data matrix of the preset image. The step of concatenating the third data matrix and the all-zero data matrix to obtain a second concatenated data matrix includes: The third data matrix, the fourth data matrix, and the all-zero data matrix are concatenated to obtain the second concatenated data matrix.
5. A computer-readable medium, characterized in that, It stores computer-readable instructions that are executed by a processor to implement some or all of the methods described in claim 1 or 2.
6. A device for image and voice matching, characterized in that, The device includes: One or more processors; and A memory storing computer-readable instructions, which, when executed, cause the processor to perform some or all of the operations of the method as described in claim 1 or 2.