Data display method and device and electronic equipment

Abstract

The invention discloses a data display method and device and electronic equipment, and belongs to the field of industrial quality detection and the technical field of image rendering. The method comprises the following steps: in response to a defect display request sent by a terminal for a target product, carrying out space alignment on a target grid model and a defect labeling depth map of the target product; the defect display request is used for requesting to display a defect position of a target product; fusing each piece of defect labeling information in the defect labeling depth map into the target grid model; rendering the target grid model to obtain a first rendered image; in the first rendering image, rendering defect positions corresponding to the defect labeling information to obtain a second rendering image; the second rendering image is sent to the terminal, the position, size, depth, protrusion and the like of the defect can be presented to a user more visually and clearly, high interactivity is achieved, and the display and analysis efficiency can be effectively improved.

Description

Technical Field

[0001] The present application belongs to the field of industrial quality inspection and image rendering technology, and in particular, relates to a data display method, device and electronic device. Background Art

[0002] In the manufacturing industry, it is very normal for products to have defects. Product defect display and defect analysis are indispensable links in reducing the defect incidence, improving product quality and production efficiency.

[0003] Related technologies display and analyze defects in products by showing two-dimensional charts such as defect annotation depth maps, tracking tables, and fishbone diagrams at defect analysis meetings. However, when the amount of data is large, the two-dimensional charts are crowded and have poor interactivity, making it impossible to intuitively display defects to users, which seriously affects the efficiency of display and analysis. Summary of the invention

[0004] The present application aims to solve at least one of the technical problems existing in the related art. To this end, the present application proposes a data display method, device and electronic device, which can more intuitively and clearly present the defect location, size, depth, protrusion, etc. to the user, has high interactivity, and can effectively improve the efficiency of display and analysis.

[0005] In a first aspect, the present application provides a data display method, applied to a server, the method comprising:

[0006] In response to a defect display request sent by the terminal for a target product, spatially aligning a target mesh model and a defect annotation depth map of the target product; the defect display request is used to request display of a defect position of the target product;

[0007] Fusing each defect annotation information in the defect annotation depth map into the target mesh model;

[0008] Rendering the target mesh model to obtain a first rendered image;

[0009] In the first rendered image, the defect position corresponding to each defect annotation information is rendered to obtain a second rendered image;

[0010] The second rendered image is sent to the terminal so as to display the second rendered image on the terminal.

[0011] According to the data display method of the present application, the target grid model and the defect annotation depth map of the target product are spatially aligned through the server; each defect annotation information in the defect annotation depth map is integrated into the target grid model, and the target grid model is rendered to obtain a first rendered image. In the first rendered image, the defect position corresponding to each defect annotation information is rendered to obtain a second rendered image; the second rendered image is sent to the terminal to solve the problems of large and complex data, insufficient spatial presentation and poor interactive experience in traditional two-dimensional tables, and can present the defect position, size, depth, protrusion, etc. to the user more intuitively and clearly, which can effectively improve the efficiency of display and analysis.

[0012] According to one embodiment of the present application, after sending the second rendered image to the terminal, the method further includes:

[0013] In response to obtaining a detail display request sent by the terminal, the detail display request is generated after the terminal detects a trigger operation of a user on a target defect position in the second rendered image, obtaining target detail information corresponding to the target defect position; the target detail information includes at least one of cause information and repair suggestion information of the target defect position;

[0014] Send target details to the terminal to display the target details on the terminal.

[0015] According to an embodiment of the present application, there is a first mapping relationship between each defect annotation information and the corresponding detail information;

[0016] After fusing each defect annotation information in the defect annotation depth map into the target mesh model, the method further includes:

[0017] Creating a second mapping relationship between each defect annotation information and a corresponding defect position in the target grid model;

[0018] Get the target detailed information corresponding to the target defect location, including:

[0019] Determining target defect annotation information corresponding to the target defect position according to a second mapping relationship between the defect annotation information and the defect position;

[0020] According to a first mapping relationship between the defect annotation information and the detail information, target detail information corresponding to the target defect annotation information is determined.

[0021] According to one embodiment of the present application, before responding to the defect display request sent by the terminal for the target product, the method further includes:

[0022] During the production process of the target product, the 3D point cloud image and fishbone diagram of the target product are obtained in real time;

[0023] Input the three-dimensional point cloud image and the fishbone diagram into the trained target neural network model, and obtain the defect annotation depth map output by the target neural network model, which includes the annotation information of each defect, and the detailed information corresponding to each defect annotation information;

[0024] A first mapping relationship between the defect annotation information and the detail information is created.

[0025] According to one embodiment of the present application, after sending the second rendered image to the terminal, the method further includes:

[0026] In response to obtaining a simulation request sent by the terminal, the simulation request is generated by the terminal after detecting a user's viewing operation on a target production line of a target product, obtaining a target grid model of the target product at a corresponding stage of the target production line;

[0027] Obtain a production line simulation model of the target production line under the target camera's viewing angle;

[0028] Deploy the target grid model of the target product at the corresponding stage of the target production line to the production line simulation model under the target camera perspective to obtain the production process simulation model of the target product on the target production line;

[0029] Render the production process simulation model to obtain a simulated production video of the target production line producing the target product;

[0030] Send simulated production video to the terminal.

[0031] According to one embodiment of the present application, before responding to a defect display request sent by the terminal for the target product, the method further includes:

[0032] Generate a target mesh model of the target product according to the three-dimensional point cloud image of the target product;

[0033] Before obtaining the production line simulation model of the target production line under the target camera viewing angle, the method further includes:

[0034] Based on the equipment models, environmental parameters and configuration parameters of each production line to which the target product belongs, a production line simulation model corresponding to each production line is generated.

[0035] According to one embodiment of the present application, the defect location is characterized by a defect area;

[0036] In the first rendered image, the defect position corresponding to each defect annotation information is rendered to obtain a second rendered image, including:

[0037] Determine the defect area corresponding to each defect annotation information in the target grid model;

[0038] Determine the defect level of each defect area according to the area size of each defect area;

[0039] According to the defect level of each defect area, each defect area is rendered to obtain a second rendered image; each defect area in the second rendered image has a corresponding rendering color, and different defect levels correspond to different rendering colors.

[0040] According to an embodiment of the present application, determining the defect area corresponding to each defect annotation information in the target grid model includes:

[0041] Determine the connected bodies in the target mesh model;

[0042] The area corresponding to the connected body is regarded as the defect area.

[0043] In a second aspect, the present application provides a data display method, applied to a terminal, the method comprising:

[0044] In response to detecting an operation of a user triggering display of a defect location of a target product, sending a defect display request for the target product to a server; the defect display request is used to request display of the defect location of the target product;

[0045] receiving a second rendered image sent by the server; the second rendered image is an image obtained by rendering the defect position corresponding to each defect annotation information in the first rendered image after the server spatially aligns the target mesh model of the target product and the defect annotation depth map of the target product, fuses each defect annotation information in the defect annotation depth map into the target mesh model, and the first rendered image is an image obtained after rendering the target mesh model;

[0046] Shows the second rendered image.

[0047] According to the data display method of the present application, in response to detecting an operation by a user to trigger the display of the defect position of the target product, a terminal sends a defect display request for the target product to a server; the defect display request is used to request the display of the defect position of the target product; a second rendered image sent by the server is received; the second rendered image is an image obtained by rendering the defect position corresponding to each defect annotation information in the first rendered image after the server spatially aligns the target grid model of the target product and the defect annotation depth map of the target product, and fuses each defect annotation information in the defect annotation depth map into the target grid model, and the first rendered image is an image obtained after rendering the target grid model; displaying the second rendered image solves the problems of large and complex data, insufficient spatial presentation, and poor interactive experience in traditional two-dimensional tables, and can more intuitively and clearly present the defect position, size, depth, protrusion, etc. to the user, thereby improving the efficiency of display and analysis.

[0048] According to one embodiment of the present application, after displaying the second rendered image, the method further includes:

[0049] In response to detecting a triggering operation of a user on a target defect position in a second rendered image, sending a detail display request to a server;

[0050] receiving target detail information of the target defect location sent by the server; the target detail information includes at least one of cause information and repair suggestion information of the target defect location;

[0051] Displays target details.

[0052] According to an embodiment of the present application, displaying target detailed information includes:

[0053] Generate a text floating box containing target details;

[0054] Display the text floating box.

[0055] According to one embodiment of the present application, after displaying the second rendered image, the method further includes:

[0056] In response to detecting a viewing operation of a target production line of a target product by a user, sending a simulation request to a server;

[0057] Obtain a simulated production video of a target production line producing a target product sent by a server;

[0058] Play the simulated production video;

[0059] The simulated production video is obtained by the server deploying the target mesh model of the target product at the corresponding stage of the target production line on the production line simulation model under the target camera perspective, obtaining the production process simulation model of the target product on the target production line, and rendering the production process simulation model.

[0060] In a third aspect, the present application provides a data display device, applied to a server, the device comprising:

[0061] A first response module is used to respond to a defect display request sent by the terminal for the target product, and spatially align the target mesh model and the defect annotation depth map of the target product; the defect display request is used to request to display the defect position of the target product;

[0062] A fusion module is used to fuse each defect annotation information in the defect annotation depth map into the target grid model;

[0063] A first rendering module, used for rendering the target mesh model to obtain a first rendered image;

[0064] A second rendering module is used to render the defect position corresponding to each defect annotation information in the first rendering image to obtain a second rendering image;

[0065] The first sending module is used to send the second rendered image to the terminal so as to display the second rendered image on the terminal.

[0066] According to the data display device of the present application, the target grid model and the defect annotation depth map of the target product are spatially aligned; each defect annotation information in the defect annotation depth map is integrated into the target grid model, and the target grid model is rendered to obtain a first rendered image. In the first rendered image, the defect position corresponding to each defect annotation information is rendered to obtain a second rendered image; the second rendered image is sent to the terminal, so as to solve the problems of large and complex data, insufficient spatial presentation and poor interactive experience in traditional two-dimensional tables, and the defect position, size, depth, protrusion, etc. can be presented to the user more intuitively and clearly, which can effectively improve the efficiency of display and analysis.

[0067] In a fourth aspect, the present application provides a data display device, applied to a terminal, the device comprising:

[0068] A first sending module is used to send a defect display request for the target product to the server in response to detecting an operation of a user triggering display of a defect location of the target product; the defect display request is used to request display of the defect location of the target product;

[0069] The first receiving module is used to receive a second rendered image sent by the server; the second rendered image is an image obtained by rendering the defect position corresponding to each defect annotation information in the first rendered image after the server spatially aligns the target mesh model of the target product and the defect annotation depth map of the target product, fuses each defect annotation information in the defect annotation depth map into the target mesh model, and the first rendered image is an image obtained after rendering the target mesh model;

[0070] The first display module is used to display the second rendered image.

[0071] According to the data display device of the present application, in response to detecting an operation by a user to trigger the display of the defect position of the target product, a defect display request for the target product is sent to a server through a terminal; the defect display request is used to request the display of the defect position of the target product; a second rendered image sent by the server is received; the second rendered image is an image obtained by rendering the defect position corresponding to each defect annotation information in the first rendered image after the server spatially aligns the target grid model of the target product and the defect annotation depth map of the target product, and fuses each defect annotation information in the defect annotation depth map into the target grid model, and the first rendered image is an image obtained after rendering the target grid model; displaying the second rendered image solves the problems of large and complex data, insufficient spatial presentation, and poor interactive experience in traditional two-dimensional tables, and can more intuitively and clearly present the defect position, size, depth, protrusion, etc. to the user, thereby improving the efficiency of display and analysis.

[0072] In a fifth aspect, the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the data display method provided in the first and second aspects described above is implemented.

[0073] In a sixth aspect, the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data display method provided in the first and second aspects above.

[0074] In a seventh aspect, the present application provides a chip, the chip comprising a processor and a communication interface, the communication interface and the processor are coupled, the processor is used to run programs or instructions to implement the data display method provided in the first and second aspects.

[0075] In an eighth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the data display method provided in the first and second aspects above.

[0076] The above one or more technical solutions in the embodiments of the present application have at least one of the following technical effects:

[0077] The target grid model and defect annotation depth map of the target product are spatially aligned; each defect annotation information in the defect annotation depth map is integrated into the target grid model, and the target grid model is rendered to obtain a first rendered image. In the first rendered image, the defect position corresponding to each defect annotation information is rendered to obtain a second rendered image; the second rendered image is sent to the terminal to solve the problems of large and complex data, insufficient spatial presentation, and poor interactive experience in traditional two-dimensional tables. The defect position, size, depth, protrusion, etc. can be presented to users more intuitively and clearly, which can effectively improve the efficiency of display and analysis.

[0078] Secondly, the pre-trained target neural network model can automatically and efficiently analyze the causes of product defects, reduce the time for manual re-judgment, and solve the time cost problem of cause analysis.

[0079] In addition, the target mesh model of the target product is automatically generated based on the 3D point cloud data, eliminating the need for manual model reconstruction of defective products and greatly reducing the workload of defective product modeling.

[0080] In addition, after receiving the simulation request, the server obtains the target grid model of the target product at the corresponding stage of the target production line, and also obtains the production line simulation model of the target production line under the target camera perspective. The target grid model of the target product at the corresponding stage of the target production line can be deployed on the production line simulation model under the target camera perspective to obtain the production process simulation model of the target product on the target production line. The production process simulation model is rendered to present the production process of the production line in three dimensions, so that the user's perspective automatically jumps to the workstation position of the production line scene to understand the link where defects occur.

[0081] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0083] Figure 1 This is one of the system architecture diagrams for implementing a data display method provided in an embodiment of the present application;

[0084] Figure 2 This is one of the flow charts of the data display method provided in the embodiment of the present application;

[0085] Figure 3 This is the second flowchart of the data display method provided in the embodiment of the present application;

[0086] Figure 4This is one of the flow diagrams of the interaction between the server and the terminal provided in the embodiment of the present application.

[0087] Figure 5 It is one of the structural schematic diagrams of the data display device provided in the embodiment of the present application;

[0088] Figure 6 This is the second structural diagram of the data display device provided in the embodiment of the present application;

[0089] Figure 7 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0090] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0091] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0092] In the manufacturing industry, the existence of defects is normal, but through effective analysis and improvement measures, the incidence of defects can be significantly reduced, and product quality and production efficiency can be improved. At present, the industry often displays product quality results in traditional two-dimensional data charts such as defect annotation depth charts, tracking tables, and fishbone diagrams. Combining these data chart tools can achieve a more comprehensive and in-depth defect analysis. For example, the defect annotation depth chart can be combined with the data from the tracking table to provide a panoramic view of the space and time of the defect occurrence. The root cause analysis of the fishbone diagram can further guide how to implement improvement measures in specific high-risk areas or production stages.

[0093] However, although data charts play an important role in analyzing and presenting information, they also have significant shortcomings. First, charts can become difficult to interpret due to data complexity and information overload. For example, when the amount of data is huge, charts may appear crowded, inconvenient to retrieve, and difficult to extract useful insights. Different users have different understandings of the design of charts. Improper design can lead to misleading presentations, which distort the actual situation of the data. Updating and maintaining charts is also a major challenge, especially when defect data is constantly generated during the production process and the data source is constantly changing, it is easy for information to become outdated or not updated in time. Even if a quality management system is introduced, the system may still face data synchronization delays, insufficient update frequency, or data processing efficiency in links that require manual intervention such as defect analysis and manual re-judgment, all of which will lead to insufficient timeliness of information. In addition, the two-dimensional representation of traditional charts also limits the display of multidimensional data, and it is even more impossible to do so in spatial display.

[0094] In addition, charts lack sufficient interactivity. When charts are used in meetings, it is difficult for participants (users) to maintain a consistent channel with the presenter in terms of visual perspective and spatial sense. Any operation on the data in the meeting can only be carried out by directing the users. The interactivity is poor and communication is difficult, which seriously affects decision-making efficiency.

[0095] To solve at least one of the above technical problems, an embodiment of the present application provides a data display method, a data display device, an electronic device and a readable storage medium.

[0096] The data display method, data display device, electronic device and readable storage medium provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0097] like Figure 1 As shown, the embodiment of the present application provides one of the system architecture diagrams for implementing a data display method, including a user 110, a terminal 120 and a server 130, wherein:

[0098] The user 110 can trigger an operation of displaying the defect location of the target product in the terminal 120, and the terminal 120 will respond to the operation by sending a defect display request for the target product to the server 130; the defect display request is used to request to display the defect location of the target product;

[0099] The server 130 spatially aligns the target mesh model and the defect annotation depth map of the target product in response to the defect display request sent by the terminal 120 for the target product;

[0100] The server 130 integrates each defect annotation information in the defect annotation depth map into the target grid model;

[0101] The server 130 renders each defect position in the first rendered image corresponding to the target grid model to obtain a second rendered image;

[0102] The server 130 sends the second rendered image to the terminal 120;

[0103] The terminal 120 receives the second rendered image sent by the server 130 and displays the second rendered image.

[0104] Terminal 120 can be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a laptop computer, a digital broadcast receiver, a MID (Mobile Internet Devices), a PDA (Personal Digital Assistant), a desktop computer, a smart home appliance, a vehicle-mounted terminal (such as a vehicle-mounted navigation terminal, a vehicle-mounted computer, etc.), a smart speaker, a smart watch, etc.

[0105] Server 130 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server or server cluster that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0106] The terminal 120 and the server 130 may establish a network connection via wired or wireless communication, but is not limited thereto.

[0107] like Figure 2 As shown, an embodiment of the present application provides one of the flow charts of a data display method, which can be applied to a server, which can be an enterprise-level high-performance server.

[0108] The data display method includes: step 210 , step 220 , step 230 , step 240 and step 250 .

[0109] Step 210: In response to a defect display request sent by the terminal for the target product, spatially align the target mesh model and the defect annotation depth map of the target product; the defect display request is used to request display of the defect position of the target product.

[0110] The target product of the embodiment of the present application may be an item produced through a manufacturing process, and the product may be a consumer product (such as household appliances) or an industrial product (a product used to produce other products or provide services, such as machinery and equipment, raw materials, and tools).

[0111] After the product is produced, it is generally necessary to conduct a quality inspection of the shell to ensure that the shell has no serious defects to avoid usage accidents or safety accidents. The embodiments of the present application mainly show the defects on the shell or appearance of the product. The types of defects include but are not limited to scratches, cracks, stains, bad pixels, shadows, uneven borders and color differences. In actual applications, the type of product defects can be determined according to the specific type of product.

[0112] The defect position in the embodiment of the present application refers to the position where the defect occurs, that is, the scratch position, crack position, stain position, bad pixel position, shadow position and color difference position, etc., and there is no limitation on this.

[0113] The target grid model corresponding to the target product is stored in the server of the embodiment of the present application. The target grid model can be generated based on the following method:

[0114] 3D laser scanners or structured light scanners are used to capture detailed geometric information on the surface of a product, generating a high-precision 3D point cloud map. The geometric information of a product includes but is not limited to at least one of its shape, size, structure, boundary, outline, and spatial position. A 3D point cloud map refers to a set of data points obtained through 3D scanning technology, which represent the shape and structure of an object or scene in 3D space. Each point is usually composed of its coordinates (x, y, z) in 3D space, as well as attribute information such as color, texture, and intensity.

[0115] Clean and repair the 3D point cloud image to reduce noise;

[0116] According to the repaired three-dimensional point cloud image, a target mesh model of the target product is generated, and the target mesh model is a three-dimensional stereo model.

[0117] It is worth noting that when collecting the above-mentioned three-dimensional point cloud image, it can be collected from multiple perspectives, and then the data collected from multiple perspectives can be aligned or registered to generate a target mesh model of the target product.

[0118] In the embodiment of the present application, the user can trigger an operation of displaying the defect location of the target product on the terminal. The operation can specifically be a click operation on the image or number of the target product, or other specific operations, which are not limited.

[0119] After detecting that the user triggers an operation to display the defect location of the target product, the terminal in the embodiment of the present application generates a defect display request, which includes the product identification of the target product (such as the product number), and is used to request the display of the defect location of the target product.

[0120] After receiving a defect display request, the server of the embodiment of the present application will obtain a target grid model of the target product and a defect annotation depth map of the target product. The defect annotation depth map may be generated and improved during the production process of the target product. The defect annotation depth map generally includes depth information of each pixel and defect annotation information of each defect.

[0121] The depth information of a pixel represents the distance from the pixel to the camera. The defect annotation information includes the defect type, defect location, defect size (defect size information, including width, height or area), defect severity (rating or classification of defects, indicating the severity or impact of the defects) and boundary information (defect outline or boundary information, usually expressed in the form of edge points or polygons).

[0122] After obtaining the target mesh model and defect annotation depth map of the target product, the server of the embodiment of the present application will spatially align the target mesh model and the defect annotation depth map, that is, align the coordinate system of the defect annotation depth map with the coordinate system of the target mesh model. After the defect annotation depth map is proportionally calculated, the obtained 2.5-dimensional coordinates are applied to the three-dimensional product model, thereby combining the depth information in the defect annotation depth map with the surface or structure of the target mesh model.

[0123] Step 220: fuse each defect annotation information in the defect annotation depth map into the target mesh model.

[0124] In the embodiment of the present application, after the target mesh model and the defect annotation depth map are spatially aligned, each defect annotation information in the defect annotation depth map is integrated into the target mesh model, so that the three-dimensional model not only displays the geometric shape of the product, but also reflects the specific location, type and severity of the defects.

[0125] Step 230: Render the target mesh model to obtain a first rendered image.

[0126] In the embodiment of the present application, a three-dimensional engine (such as the open graphics library OpenGL) is deployed in the server, which can build scenes and set camera perspectives, lighting, and interactors.

[0127] After integrating each defect annotation information in the defect annotation depth map into the target mesh model, the server can render the target mesh model of the target product according to the three-dimensional engine to obtain a first rendered image corresponding to the three-dimensional model of the target product.

[0128] Step 240: Render the defect positions corresponding to the defect annotation information in the first rendered image to obtain a second rendered image.

[0129] It can be understood that each defect annotation information has a corresponding defect position. The first rendered image is obtained after rendering the fused target mesh model, and the defect position in the target mesh model is also rendered. However, since the color of the defect position itself may be closer to the original color of the target product, or due to the influence of rendering parameters (saturation, brightness), the defect position in the first rendered image may not be particularly obvious. Therefore, in order to highlight the defect position in the first rendered image, the defect position corresponding to each defect annotation information can be rendered to obtain a second rendered image. The defect positions in the second rendered image are more obvious, such as marking each defect position in red.

[0130] Step 250: Send the second rendered image to the terminal to display the second rendered image on the terminal.

[0131] After obtaining the second rendered image, the server in the embodiment of the present application can send the second rendered image to the terminal, so that the terminal can visualize the second rendered image, making it convenient to intuitively and clearly view the defect locations of the target product.

[0132] The second rendered image presents a three-dimensional display effect. Users can rotate, zoom, and translate the second rendered image in the terminal. It has high interactivity, allowing users to view and analyze defects from multiple angles to ensure more accurate positioning and understanding of the defects.

[0133] The embodiment of the present application spatially aligns a target grid model and a defect annotation depth map of a target product; fuses each defect annotation information in the defect annotation depth map into the target grid model, renders the target grid model to obtain a first rendered image, renders the defect position corresponding to each defect annotation information in the first rendered image to obtain a second rendered image; and sends the second rendered image to a terminal, thereby solving the problems of large and complex data, insufficient spatial presentation, and poor interactive experience in traditional two-dimensional tables, and can present defect positions, sizes, depths, protrusions, etc. to users more intuitively and clearly, thereby effectively improving the efficiency of display and analysis.

[0134] In some embodiments, after sending the second rendered image to the terminal, the method further includes:

[0135] In response to obtaining a detail display request sent by the terminal, the detail display request is generated after the terminal detects a trigger operation of a user on a target defect position in the second rendered image, obtaining target detail information corresponding to the target defect position; the target detail information includes at least one of cause information and repair suggestion information of the target defect position;

[0136] Send target details to the terminal to display the target details on the terminal.

[0137] The embodiment of the present application also supports registering a defect click event, which can be a trigger operation of the user on the target defect location.

[0138] When the user intends to view the detailed information of a target defect position, the user can trigger the target defect position in the second rendered image. After detecting the user's trigger operation on the target defect position in the second rendered image, the terminal generates a detail display request and sends the detail display request to the server. The detail display request includes a location identifier of the target defect position, and the detail display request is used to request to view the target detailed information of the target defect position. The target detailed information includes at least one of the cause information and repair suggestion information of the target defect position.

[0139] After receiving the detail display request sent by the terminal, the server in the embodiment of the present application sends the target detail information to the terminal, so as to display the target detail information on the terminal.

[0140] The target detailed information and defect annotation depth map of the embodiment of the present application can be obtained through a machine learning model. Specifically, during the production process of the target product, a three-dimensional point cloud map and a fishbone diagram of the target product are obtained in real time. The fishbone diagram (also known as a cause-and-effect diagram or Ishikawa diagram) is a tool for identifying and analyzing the root causes of problems, and is commonly used in product quality management and process improvement. The fishbone diagram usually includes the following main components:

[0141] Trunk: The trunk of a fishbone diagram represents the problem or effect to be analyzed and is usually shown as a box on the right side of the diagram.

[0142] Main categories: The main bones extending from the trunk represent the main categories that affect the problem. There are usually six common categories: People (human factors), Machines (equipment and technology), Materials (raw materials and components), Methods (processes and operating methods), Measurement (measurement and evaluation tools), Environment (external environmental factors)

[0143] Secondary causes: Below the main skeleton, you can continue to add more detailed secondary causes to help drill down into the influencing factors of each main category.

[0144] Arrows and connections: Each bone and secondary cause is usually connected to the main bone with an arrow to clearly show the cause and effect relationship.

[0145] Problem Description: On the right side of the fishbone diagram, clearly describe the problem or goal to be solved in order to focus the analysis.

[0146] During the model training stage, a large number of training samples can be collected. Each training sample includes a sample 3D point cloud map and a sample fishbone diagram corresponding to a sample product. The training label of each training sample is a defect annotation depth map of the defect annotation information, and detailed information corresponding to each defect annotation information.

[0147] A massive amount of training samples can be input into the target neural network model to train the target neural network model until the training end conditions are met to obtain a trained target neural network model. During the training process, the target neural network model can perform preprocessing and feature extraction on each training sample to identify defects.

[0148] The target neural network model of the embodiment of the present application is a machine learning model. The training and use of the machine learning model can automatically and efficiently analyze the causes of product defects, reduce the time for manual re-judgment, and solve the time cost problem of cause analysis.

[0149] In addition, the above process of obtaining defect annotation information and detailed information can be executed locally by the server performing the rendering operation, or by the server on the production line. When executed by the server on the production line, the effect of edge computing can be achieved, which can reduce the workload of the rendering server.

[0150] In some embodiments, there is a first mapping relationship between each defect annotation information and the corresponding detail information;

[0151] After fusing each defect annotation information in the defect annotation depth map into the target mesh model, the method further includes:

[0152] Creating a second mapping relationship between each defect annotation information and a corresponding defect position in the target grid model;

[0153] Get the target detailed information corresponding to the target defect location, including:

[0154] Determining target defect annotation information corresponding to the target defect position according to a second mapping relationship between the defect annotation information and the defect position;

[0155] According to a first mapping relationship between the defect annotation information and the detail information, target detail information corresponding to the target defect annotation information is determined.

[0156] After obtaining each defect annotation information and corresponding detail information, the embodiment of the present application creates a first mapping relationship between each defect annotation information and the corresponding detail information.

[0157] In addition, after integrating each defect annotation information in the defect annotation depth map into the target mesh model, the embodiment of the present application also creates a second mapping relationship between the defect annotation information and the corresponding defect position in the target mesh model.

[0158] Therefore, when obtaining the target detailed information corresponding to the target defect position as mentioned above, specifically, the target defect labeling information corresponding to the target defect position can be determined based on the second mapping relationship between the defect labeling information and the defect position; and the target detailed information corresponding to the target defect labeling information can be determined based on the first mapping relationship between the defect labeling information and the detailed information.

[0159] In some embodiments, before responding to the defect display request sent by the terminal for the target product, the method further includes:

[0160] During the production process of the target product, the 3D point cloud image and fishbone diagram of the target product are obtained in real time;

[0161] Input the three-dimensional point cloud image and the fishbone diagram into the trained target neural network model, and obtain the defect annotation depth map output by the target neural network model, which includes the annotation information of each defect, and the detailed information corresponding to each defect annotation information;

[0162] A first mapping relationship between the defect annotation information and the detail information is created.

[0163] This has been explained in the above-mentioned embodiment and will not be described again here.

[0164] In some embodiments, after sending the second rendered image to the terminal, the method further includes:

[0165] In response to obtaining a simulation request sent by the terminal, the simulation request is generated by the terminal after detecting a user's viewing operation on a target production line of a target product, obtaining a target grid model of the target product at a corresponding stage of the target production line;

[0166] Obtain a production line simulation model of the target production line under the target camera's viewing angle;

[0167] Deploy the target grid model of the target product at the corresponding stage of the target production line to the production line simulation model under the target camera perspective to obtain the production process simulation model of the target product on the target production line;

[0168] Render the production process simulation model to obtain a simulated production video of the target production line producing the target product;

[0169] Send simulated production video to the terminal.

[0170] The embodiment of the present application also supports product production process traceability. The terminal can provide users with multiple production line display controls. The user can click the control corresponding to the target production line, thereby triggering a viewing operation on the target production line of the target product. When the terminal detects the user's viewing operation on the target production line of the target product, it generates a simulation request, which is used to request a simulation of the production process of the target product on the target production line.

[0171] The target production line may be a production line of any node, or a production line where a certain defect occurs, without limitation.

[0172] After receiving the simulation request, the server obtains the target grid model of the target product at the corresponding stage of the target production line, and also obtains the production line simulation model of the target production line under the target camera perspective. The target camera perspective refers to the perspective of the camera capturing images.

[0173] The target grid model of the target product at the corresponding stage of the target production line can be deployed on the production line simulation model under the target camera perspective to obtain a production process simulation model of the target product on the target production line, thereby presenting the production process of the production line in a three-dimensional manner.

[0174] The production process simulation model is a dynamic simulation process. Therefore, the production process simulation model can be rendered to obtain a simulated production video of the target production process, and a simulated production video of the target production line producing the target product can be sent to the terminal, so that the user's perspective automatically jumps to the workstation position of the production line scene to understand the link where defects occur.

[0175] In actual application, the GPU or CPU can be used for high-performance computing and drawing and rendering the production line model and the target mesh model of the target product.

[0176] In some embodiments, before responding to the defect display request sent by the terminal for the target product, the method further includes:

[0177] Generate a target mesh model of the target product according to the three-dimensional point cloud image of the target product;

[0178] Before obtaining the production line simulation model of the target production line under the target camera viewing angle, the method further includes:

[0179] Based on the equipment models, environmental parameters and configuration parameters of each production line to which the target product belongs, a production line simulation model corresponding to each production line is generated.

[0180] In the production process of the target product, the embodiment of the present application can obtain a three-dimensional point cloud image of the target product in real time and generate a target mesh model of the target product;

[0181] It is also possible to generate production line simulation models corresponding to each production line based on the equipment models, environmental parameters and configuration parameters of each production line to which the target product belongs. The detailed process of generating the production line simulation model can be found in the prior art and will not be repeated here.

[0182] The embodiment of the present application automatically generates a target mesh model of the target product based on three-dimensional point cloud data, defect annotation depth map, etc., without the need for manual model reconstruction of the defective product, thereby greatly reducing the workload of defective product modeling.

[0183] In some embodiments, the defect location is characterized by a defect area;

[0184] In the first rendered image, the defect position corresponding to each defect annotation information is rendered to obtain a second rendered image, including:

[0185] Determine the defect area corresponding to each defect annotation information in the target grid model;

[0186] Determine the defect level of each defect area according to the area size of each defect area;

[0187] According to the defect level of each defect area, each defect area is rendered to obtain a second rendered image; each defect area in the second rendered image has a corresponding rendering color, and different defect levels correspond to different rendering colors.

[0188] The embodiment of the present application can determine the defect level of the defect area according to the area size of the defect area. Generally, the larger the defect area, the higher the defect level. Each defect area can be rendered according to its defect level to obtain a second rendered image; each defect area in the second rendered image has a corresponding rendering color, and different defect levels correspond to different rendering colors. For example, for a defect area with a higher defect level, a dark red color can be set, for a defect area with a medium defect level, a light red color can be set, and for a defect area with a lower defect level, a yellow color can be set, etc.

[0189] The embodiment of the present application renders defect areas of different defect levels by using different rendering colors, so as to intuitively inform the user of the defect level of each defect area.

[0190] In some embodiments, determining the defect area corresponding to each defect annotation information in the target grid model includes:

[0191] Determine the connected bodies in the target mesh model;

[0192] The area corresponding to the connected body is regarded as the defect area.

[0193] Specifically, each connected body in the target grid model can be determined. A connected body is an area composed of defect points and connected to each other in space. Each connected body corresponds to a coherent and closed coordinate area. Therefore, the area corresponding to the connected body can be used as a defect area.

[0194] like Figure 3 As shown, an embodiment of the present application provides a second flowchart of a data display method, which can be applied to a terminal. Through the websocket technology, the terminal and the server can maintain real-time communication and can drive the update of the front-end data through instructions.

[0195] The data display method includes the following steps 310, 320 and 330:

[0196] Step 310: In response to detecting a user triggering an operation to display a defect location of a target product, a defect display request for the target product is sent to a server; the defect display request is used to request display of the defect location of the target product.

[0197] Step 320, receiving a second rendered image sent by the server; the second rendered image is an image obtained by rendering the defect position corresponding to each defect annotation information in the first rendered image after the server spatially aligns the target mesh model of the target product and the defect annotation depth map of the target product, fuses each defect annotation information in the defect annotation depth map into the target mesh model, and the first rendered image is an image obtained after rendering the target mesh model.

[0198] Step 330: Display the second rendered image.

[0199] After the product is produced, it is generally necessary to conduct a quality inspection of the shell to ensure that the shell has no serious defects to avoid usage accidents or safety accidents. The embodiments of the present application mainly show the defects on the shell or appearance of the product. The types of defects include but are not limited to scratches, cracks, stains, bad pixels, shadows, uneven borders and color differences. In actual applications, the type of product defects can be determined according to the specific type of product.

[0200] The defect position in the embodiment of the present application refers to the position where the defect occurs, that is, the scratch position, crack position, stain position, bad pixel position, shadow position and color difference position, etc., and there is no limitation on this.

[0201] In the embodiment of the present application, the user can trigger an operation of displaying the defect location of the target product on the terminal. The operation can specifically be a click operation on the image or number of the target product, or other specific operations, which are not limited.

[0202] After detecting that a user triggers an operation to display the defect location of a target product, the terminal in the embodiment of the present application generates a defect display request and sends the defect display request to a server. The defect display request includes a product identifier (such as a product number) of the target product, and the defect display request is used to request display of the defect location of the target product.

[0203] After receiving the defect display request, the server of the embodiment of the present application spatially aligns the target mesh model of the target product and the defect annotation depth map of the target product, fuses each defect annotation information in the defect annotation depth map into the target mesh model, and then renders the defect position corresponding to each defect annotation information in the first rendered image to obtain an image. The first rendered image is an image obtained after rendering the target mesh model. The detailed process is shown in FIG. Figure 1 The detailed embodiments in the description will not be repeated here.

[0204] After receiving the second rendered image, the terminal in the embodiment of the present application displays the second rendered image, thereby more intuitively showing the user the locations of various defects existing in the target grid model of the target product.

[0205] In addition, the menu editor includes pre-set parameters. Users can adjust the parameters to change the scene light source, color temperature, etc. and observe their impact on the model.

[0206] In the embodiment of the present application, in response to detecting an operation triggered by a user to display the defect position of the target product, the terminal sends a defect display request for the target product to the server; the defect display request is used to request the display of the defect position of the target product; and a second rendered image sent by the server is received; the second rendered image is an image obtained by rendering the defect position corresponding to each defect annotation information in the first rendered image after the server spatially aligns the target grid model of the target product and the defect annotation depth map of the target product, and fuses each defect annotation information in the defect annotation depth map into the target grid model, and the first rendered image is an image obtained after rendering the target grid model; displaying the second rendered image solves the problems of large and complex data, insufficient spatial presentation, and poor interactive experience in traditional two-dimensional tables, and can more intuitively and clearly present the defect position, size, depth, protrusion, etc. to the user, thereby improving the efficiency of display and analysis.

[0207] In some embodiments, after presenting the second rendered image, the method further comprises:

[0208] In response to detecting a triggering operation of a user on a target defect position in a second rendered image, sending a detail display request to a server;

[0209] receiving target detail information of the target defect location sent by the server; the target detail information includes at least one of cause information and repair suggestion information of the target defect location;

[0210] Displays target details.

[0211] The embodiment of the present application also supports registering a defect click event, which can be a trigger operation of the user on the target defect location.

[0212] When the user intends to view the detailed information of a target defect position, the user can trigger the target defect position in the second rendered image. After detecting the user's trigger operation on the target defect position in the second rendered image, the terminal generates a detail display request and sends the detail display request to the server. The detail display request includes a location identifier of the target defect position, and the detail display request is used to request to view the target detailed information of the target defect position. The target detailed information includes at least one of the cause information and repair suggestion information of the target defect position.

[0213] After receiving the detail display request sent by the terminal, the server in the embodiment of the present application sends the target detail information to the terminal, and after receiving the target detail information, the terminal displays the target detail information.

[0214] In some embodiments, the target detailed information is displayed, including:

[0215] Generate a text floating box containing target details;

[0216] Display the text floating box.

[0217] The embodiment of the present application can present at least one of the cause information and repair suggestion information of the target defect location obtained through the machine learning model through a text floating box, so that more content can be displayed in a limited interface space without stacking too much information in the interface, keeping the interface neat.

[0218] In some embodiments, after presenting the second rendered image, the method further comprises:

[0219] In response to detecting a viewing operation of a target production line of a target product by a user, sending a simulation request to a server;

[0220] Obtain a simulated production video of a target production line producing a target product sent by a server;

[0221] Play the simulated production video;

[0222] The simulated production video is obtained by the server deploying the target mesh model of the target product at the corresponding stage of the target production line on the production line simulation model under the target camera perspective, obtaining the production process simulation model of the target product on the target production line, and rendering the production process simulation model.

[0223] The embodiment of the present application also supports product production process traceability. The terminal can provide users with multiple production line display controls. The user can click the control corresponding to the target production line, thereby triggering a viewing operation on the target production line of the target product. When the terminal detects the user's viewing operation on the target production line of the target product, it generates a simulation request, which is used to request a simulation of the production process of the target product on the target production line.

[0224] The target production line may be a production line of any node, or a production line where a certain defect occurs, without limitation.

[0225] After receiving the simulation request, the server obtains the target grid model of the target product at the corresponding stage of the target production line, and also obtains the production line simulation model of the target production line under the target camera perspective. The target camera perspective refers to the perspective from which the camera captures images. The target grid model of the target product at the corresponding stage of the target production line can be deployed on the production line simulation model under the target camera perspective to obtain a production process simulation model of the target product on the target production line, thereby presenting the production process of the production line in three dimensions.

[0226] The production process simulation model is a dynamic simulation process. Therefore, the production process simulation model can be rendered to obtain a simulated production video of the target production process, and a simulated production video of the target production line producing the target product can be sent to the terminal. The simulated production video can be played so that the user's perspective automatically jumps to the workstation position of the production line scene to understand the link where defects occur.

[0227] The embodiment of the present application supports a single-user instance function, which refers to a mode in which the software or application only supports one user at a time. Through the single-user instance function, different users are allowed to share operations on the same page, and data is synchronously shared to other terminals. The operation sharing function further enhances the clarity of team collaboration and information transmission.

[0228] In addition, the embodiments of the present application also support a multi-user instance function. The multi-user instance function refers to a mode in which the software or application can support simultaneous use by multiple users. The multi-user instance function is used to release the sharing operation.

[0229] like Figure 4 As shown, the embodiment of the present application provides one of the flow diagrams of interaction between a server and a terminal, including the following steps:

[0230] Step 401: In response to detecting a user triggering an operation to display a defect location of a target product, the terminal sends a defect display request for the target product to a server; the defect display request is used to request display of the defect location of the target product;

[0231] Step 402: The server spatially aligns a target mesh model and a defect annotation depth map of the target product in response to a defect display request sent by the terminal for the target product.

[0232] Step 403: The server integrates each defect annotation information in the defect annotation depth map into the target grid model;

[0233] Step 404: The server renders the target grid model to obtain a first rendered image;

[0234] Step 405: The server renders the defect position corresponding to each defect annotation information in the first rendered image to obtain a second rendered image;

[0235] Step 406: The server sends a second rendered image to the terminal;

[0236] Step 407: The terminal displays the second rendered image;

[0237] Step 408: In response to detecting a trigger operation of the user on the target defect position in the second rendered image, the terminal sends a detail display request to the server;

[0238] Step 409: The server obtains target detail information corresponding to the target defect position in response to the detail display request sent by the terminal; the target detail information includes at least one of cause information and repair suggestion information of the target defect position;

[0239] Step 410: The server sends target details to the terminal;

[0240] Step 411: The terminal generates a text floating box containing target detailed information and displays the text floating box.

[0241] Step 412: In response to detecting the user's viewing operation on the target production line of the target product, the terminal sends a simulation request to the server;

[0242] Step 413: The server obtains a target grid model of the target product at a corresponding stage of the target production line;

[0243] Step 414: The server obtains a production line simulation model of the target production line under the viewing angle of the target camera;

[0244] Step 415: The server deploys the target grid model of the target product at the corresponding stage of the target production line to the production line simulation model under the target camera viewing angle, and obtains the production process simulation model of the target product on the target production line;

[0245] Step 416: The server renders the production process simulation model to obtain a simulated production video of the target production line producing the target product;

[0246] Step 417: The server sends the simulated production video to the terminal;

[0247] Step 418: The terminal plays the simulated production video.

[0248] The detailed implementation process of the above steps 401 to 418 is shown in the above embodiments and will not be described again here.

[0249] The data display method provided in the embodiment of the present application can be executed by a data display device. In the embodiment of the present application, the data display device provided in the embodiment of the present application is described by taking the data display method executed by the data display device as an example.

[0250] The embodiment of the present application also provides a data display device, which is applied to a server.

[0251] like Figure 5 As shown, the data display device 50 includes: a first response module 510, a fusion module 520, a first rendering module 530, a second rendering module 540 and a first sending module 550.

[0252] The first response module 510 is used to spatially align the target mesh model and the defect annotation depth map of the target product in response to a defect display request sent by the terminal for the target product; the defect display request is used to request to display the defect position of the target product;

[0253] A fusion module 520 is used to fuse each defect annotation information in the defect annotation depth map into the target mesh model;

[0254] A first rendering module 530, configured to render the target mesh model to obtain a first rendered image;

[0255] A second rendering module 540 is used to render the defect position corresponding to each defect annotation information in the first rendering image to obtain a second rendering image;

[0256] The first sending module 550 is configured to send the second rendered image to the terminal so as to display the second rendered image on the terminal.

[0257] According to the data display device 50 provided in the embodiment of the present application, the target grid model and the defect annotation depth map of the target product are spatially aligned; each defect annotation information in the defect annotation depth map is integrated into the target grid model, and the target grid model is rendered to obtain a first rendered image. In the first rendered image, the defect position corresponding to each defect annotation information is rendered to obtain a second rendered image; the second rendered image is sent to the terminal to solve the problems of large and complex data, insufficient spatial presentation, and poor interactive experience in traditional two-dimensional tables, and the defect position, size, depth, protrusion, etc. can be presented to the user more intuitively and clearly, which can effectively improve the efficiency of display and analysis.

[0258] In some embodiments, the data display device 50 further includes:

[0259] a second response module, configured to respond to a detail display request sent by the terminal, the detail display request being generated after the terminal detects a trigger operation of a user on a target defect position in the second rendered image, and obtain target detail information corresponding to the target defect position; the target detail information includes at least one of cause information and repair suggestion information of the target defect position;

[0260] The second sending module is used to send the target detailed information to the terminal so as to display the target detailed information on the terminal.

[0261] In some embodiments, there is a first mapping relationship between each defect annotation information and the corresponding detail information;

[0262] The data display device 50 further includes:

[0263] A first creation module is used to create a second mapping relationship between each defect annotation information and a corresponding defect position in the target grid model;

[0264] The second response module is specifically used for:

[0265] Determining target defect annotation information corresponding to the target defect position according to a second mapping relationship between the defect annotation information and the defect position;

[0266] According to a first mapping relationship between the defect annotation information and the detail information, target detail information corresponding to the target defect annotation information is determined.

[0267] In some embodiments, the data display device 50 further includes:

[0268] The first acquisition module is used to acquire the three-dimensional point cloud image and fishbone diagram of the target product in real time during the production process of the target product;

[0269] An input module, used to input the three-dimensional point cloud image and the fishbone diagram into the trained target neural network model, and obtain a defect annotation depth map output by the target neural network model containing the annotation information of each defect, and detailed information corresponding to each defect annotation information;

[0270] The second creation module is used to create a first mapping relationship between the defect annotation information and the detail information.

[0271] In some embodiments, the data display device 50 further includes:

[0272] A third corresponding module is used to obtain a target grid model of the target product at a corresponding stage of the target production line in response to a simulation request sent by the terminal, the simulation request being generated by the terminal after detecting a user's viewing operation on the target production line of the target product;

[0273] The second acquisition module is used to acquire a production line simulation model of the target production line under the viewing angle of the target camera;

[0274] A deployment module is used to deploy the target grid model of the target product at the corresponding stage of the target production line to the production line simulation model under the target camera perspective to obtain a production process simulation model of the target product on the target production line;

[0275] The third rendering module is used to render the production process simulation model to obtain a simulated production video of the target production line producing the target product;

[0276] The third sending module is used to send the simulated production video to the terminal.

[0277] In some embodiments, the data display device 50 further includes:

[0278] A first generation module is used to generate a target mesh model of a target product according to a three-dimensional point cloud image of the target product;

[0279] The second generation module is used to generate a production line simulation model corresponding to each production line based on the equipment model, environmental parameters and configuration parameters of each production line to which the target product belongs.

[0280] In some embodiments, the defect location is characterized by a defect area;

[0281] The second rendering module 540 includes:

[0282] The region determination submodule is used to determine the defect region corresponding to each defect annotation information in the target grid model;

[0283] A level determination submodule, used to determine the defect level of each defect area according to the area size of each defect area;

[0284] The rendering submodule is used to render each defect area according to the defect level of each defect area to obtain a second rendered image; each defect area in the second rendered image has a corresponding rendering color, and different defect levels correspond to different rendering colors.

[0285] In some embodiments, the region determination submodule is specifically used to:

[0286] Determine the connected bodies in the target mesh model;

[0287] The area corresponding to the connected body is regarded as the defect area.

[0288] The embodiment of the present application also provides a data display device, which is applied to a terminal.

[0289] like Figure 6 As shown, the data display device 60 includes: a first sending module 610 , a first receiving module 620 and a first display module 630 .

[0290] The first sending module 610 is used to send a defect display request for the target product to the server in response to detecting an operation of the user triggering the display of the defect location of the target product; the defect display request is used to request the display of the defect location of the target product;

[0291] The first receiving module 620 is used to receive a second rendered image sent by the server; the second rendered image is an image obtained by rendering the defect position corresponding to each defect annotation information in the first rendered image after the server spatially aligns the target mesh model of the target product and the defect annotation depth map of the target product, fuses each defect annotation information in the defect annotation depth map into the target mesh model, and the first rendered image is an image obtained after rendering the target mesh model;

[0292] The first display module 630 is used to display the second rendered image.

[0293] The data display device provided in the embodiment of the present application sends a defect display request for the target product to the server in response to detecting an operation of a user triggering the display of the defect position of the target product through a terminal; the defect display request is used to request the display of the defect position of the target product; and a second rendered image sent by the server is received; the second rendered image is an image obtained by rendering the defect position corresponding to each defect annotation information in the first rendered image after the server spatially aligns the target grid model of the target product and the defect annotation depth map of the target product, and fuses each defect annotation information in the defect annotation depth map into the target grid model, and the first rendered image is an image obtained after rendering the target grid model; displaying the second rendered image solves the problems of large and complex data, insufficient spatial presentation, and poor interactive experience in traditional two-dimensional tables, and can more intuitively and clearly present the defect position, size, depth, protrusion, etc. to the user, thereby improving the efficiency of display and analysis.

[0294] In some embodiments, the data display device 60 further includes:

[0295] A second sending module, configured to send a detail display request to a server in response to detecting a triggering operation of a user on a target defect position in a second rendered image;

[0296] The second receiving module is used to receive target detail information of the target defect position sent by the server; the target detail information includes at least one of cause information and repair suggestion information of the target defect position;

[0297] The second display module is used to display target detailed information.

[0298] In some embodiments, the second display module is specifically used for:

[0299] Generate a text floating box containing target details;

[0300] Display the text floating box.

[0301] In some embodiments, the data display device 60 further includes:

[0302] A third sending module, configured to send a simulation request to a server in response to detecting a user's viewing operation on a target production line of a target product;

[0303] The third receiving module is used to obtain the simulated production video of the target production line producing the target product sent by the server;

[0304] Playback module, used to play the simulated production video;

[0305] The simulated production video is obtained by the server deploying the target mesh model of the target product at the corresponding stage of the target production line on the production line simulation model under the target camera perspective, obtaining the production process simulation model of the target product on the target production line, and rendering the production process simulation model.

[0306] The data display device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a car electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtualreality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personalcomputer, PC), a television (television, TV), a teller machine or a self-service machine, etc., and the embodiment of the present application is not specifically limited.

[0307] The data display device in the embodiment of the present application may be a device having an operating system. The operating system may be a Microsoft (Windows) operating system, an Android (Android) operating system, an IOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0308] The data display device provided in the embodiment of the present application can achieve Figures 1 to 4 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0309] In some embodiments, Figure 7 As shown, an embodiment of the present application also provides an electronic device 700, including a processor 701, a memory 702, and a computer program stored in the memory 702 and executable on the processor 701. When the program is executed by the processor 701, each process of the above-mentioned data display method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0310] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0311] An embodiment of the present application also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned data display method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0312] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0313] An embodiment of the present application also provides a computer program product, including a computer program, which implements the above-mentioned data display method when executed by a processor.

[0314] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0315] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned data display method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0316] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0317] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0318] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the relevant technology, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0319] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

[0320] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0321] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

Claims

1. A data display method, characterized in that: Applicable to servers, including: In response to a defect display request sent by a terminal for a target product, spatially aligning a target mesh model and a defect annotation depth map of the target product; the defect display request is used to request display of a defect position of the target product; Merging each defect annotation information in the defect annotation depth map into the target grid model; Rendering the target mesh model to obtain a first rendered image; In the first rendered image, the defect position corresponding to each defect annotation information is rendered to obtain a second rendered image; The second rendered image is sent to the terminal so as to display the second rendered image on the terminal.

2. The data display method according to claim 1, characterized in that: After sending the second rendered image to the terminal, the method further includes: In response to obtaining a detail display request sent by the terminal, the detail display request is generated after the terminal detects a trigger operation of a user on a target defect position in the second rendered image, obtaining target detail information corresponding to the target defect position; the target detail information includes at least one of cause information and repair suggestion information of the target defect position; The target detailed information is sent to the terminal so as to display the target detailed information on the terminal.

3. The data display method according to claim 2, characterized in that: There is a first mapping relationship between each defect annotation information and the corresponding detail information; After fusing each defect annotation information in the defect annotation depth map into the target grid model, the method further includes: Creating a second mapping relationship between each of the defect annotation information and a corresponding defect position in the target grid model; The obtaining of target detailed information corresponding to the target defect position includes: Determining target defect annotation information corresponding to the target defect position according to a second mapping relationship between the defect annotation information and the defect position; According to a first mapping relationship between the defect annotation information and the detail information, target detail information corresponding to the target defect annotation information is determined.

4. The data display method according to claim 3, characterized in that: Before responding to the defect display request sent by the terminal for the target product, the method further includes: During the production process of the target product, a three-dimensional point cloud image and a fishbone diagram of the target product are obtained in real time; Inputting the three-dimensional point cloud image and the fishbone diagram into a trained target neural network model, obtaining a defect annotation depth map output by the target neural network model that includes each defect annotation information, and detailed information corresponding to each defect annotation information; A first mapping relationship between the defect annotation information and the detail information is created.

5. The data display method according to claim 4, characterized in that: After sending the second rendered image to the terminal, the method further includes: In response to obtaining a simulation request sent by a terminal, the simulation request is generated by the terminal after detecting a user's viewing operation on a target production line of the target product, obtaining a target grid model of the target product at a corresponding stage of the target production line; Acquire a production line simulation model of the target production line under the viewing angle of a target camera; Deploy the target grid model of the target product at the corresponding stage of the target production line to the production line simulation model under the target camera viewing angle to obtain a production process simulation model of the target product on the target production line; Rendering the production process simulation model to obtain a simulated production video of the target production line producing the target product; The simulated production video is sent to the terminal.

6. The data display method according to claim 5, characterized in that: Before responding to the defect display request sent by the terminal for the target product, the method further includes: Generating a target mesh model of the target product according to the three-dimensional point cloud image of the target product; Before obtaining the production line simulation model of the target production line under the target camera viewing angle, the method further includes: Based on the equipment models, environmental parameters and configuration parameters of each production line to which the target product belongs, a production line simulation model corresponding to each production line is generated.

7. The data display method according to any one of claims 1 to 6, characterized in that: The defect location is characterized by a defect area; The step of rendering the defect positions corresponding to the defect annotation information in the first rendered image to obtain a second rendered image includes: Determine the defect area corresponding to each defect annotation information in the target grid model; Determine the defect level of each defect area according to the area size of each defect area; According to the defect level of each defect area, each defect area is rendered to obtain a second rendered image; each defect area in the second rendered image has a corresponding rendering color, and different defect levels correspond to different rendering colors.

8. The data display method according to claim 7, characterized in that: The step of determining the defect area corresponding to each defect annotation information in the target grid model includes: Determining a connected body in the target grid model; The area corresponding to the connected body is taken as the defect area.

9. A data display method, characterized in that: Applied to terminals, including: In response to detecting an operation of a user triggering display of a defect location of a target product, sending a defect display request for the target product to a server; the defect display request is used to request display of the defect location of the target product; receiving a second rendered image sent by the server; the second rendered image is an image obtained by rendering the defect position corresponding to each defect annotation information in the first rendered image after the server spatially aligns the target mesh model of the target product and the defect annotation depth map of the target product, fuses each defect annotation information in the defect annotation depth map into the target mesh model, and the first rendered image is an image obtained after rendering the target mesh model; The second rendered image is displayed.

10. The data display method according to claim 9, characterized in that: After displaying the second rendered image, the method further includes: In response to detecting a triggering operation of a user on a target defect position in the second rendered image, sending a detail display request to the server; receiving target detail information of the target defect position sent by a server; the target detail information includes at least one of cause information and repair suggestion information of the target defect position; The target detailed information is displayed.

11. The data display method according to claim 10, characterized in that: The displaying of the target detailed information includes: Generate a text floating box containing detailed information of the target; The text floating box is displayed.

12. The data display method according to claim 9, characterized in that: After displaying the second rendered image, the method further includes: In response to detecting a viewing operation of a user on a target production line of the target product, sending a simulation request to a server; Acquire a simulated production video of the target production line producing the target product, which is sent by the server; Playing the simulated production video; The simulated production video is obtained by the server deploying the target grid model of the target product at the corresponding stage of the target production line on the production line simulation model under the target camera perspective, obtaining the production process simulation model of the target product on the target production line, and rendering the production process simulation model.

13. A data display device, characterized in that: Applicable to servers, including: A first response module, configured to spatially align a target mesh model and a defect annotation depth map of a target product in response to a defect display request sent by a terminal for the target product; the defect display request is used to request display of a defect position of the target product; A fusion module, used to fuse each defect annotation information in the defect annotation depth map into the target grid model; A first rendering module, used for rendering the target mesh model to obtain a first rendered image; A second rendering module, configured to render the defect position corresponding to each defect annotation information in the first rendering image to obtain a second rendering image; The first sending module is used to send the second rendered image to the terminal, so as to display the second rendered image on the terminal.

14. A data display device, characterized in that: Applied to terminals, including: A first sending module is used to send a defect display request for the target product to a server in response to detecting an operation of a user triggering display of a defect location of the target product; the defect display request is used to request display of the defect location of the target product; a first receiving module, configured to receive a second rendered image sent by the server; the second rendered image is an image obtained by rendering the defect position corresponding to each defect annotation information in the first rendered image after the server spatially aligns the target mesh model of the target product and the defect annotation depth map of the target product, fuses each defect annotation information in the defect annotation depth map into the target mesh model, and the first rendered image is an image obtained after rendering the target mesh model; The first display module is used to display the second rendered image.

15. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the data display method as described in any one of claims 1-8 and 9-12 is implemented.

16. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data display method as described in any one of claims 1-8 and 9-12 is implemented.

17. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the data display method according to any one of claims 1 to 8 and 9 to 12 is implemented.

Images