Photo-based interface for accessing position-based engineering information

Abstract

To provide a method for creating a photo-based interface for displaying accurate space data for analyzing a structure having a regulated coordinate system, a system and a non-temporary computer readable medium.SOLUTION: A method includes generating a photographic record showing a component, retrieving one or more technical documents, and extracting primary coordinate data of the component. The primary coordinate data is integrated into the photographic record, wherein a reference plane is defined in the photographic record based on the primary coordinate data, one or more sub-components in the defined reference plane are identified, and secondary coordinate data of each of the one or more sub-components is generated.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] Aspects of the present disclosure relate to spatial data mapping, and more particularly to creating a photo-based interface that displays accurate spatial data for analyzing structures having a defined coordinate system. [Background technology]

[0002] Traditionally, identifying nonconformities within vehicle components involves examining a wide range of technical resources, including detailed engineering drawings, installation plans, and three-dimensional models, among others. Using these documents to determine the precise location of a specific instrument not only requires substantial technical knowledge but also presents other challenges. For example, it requires extensive cross-referencing and correlation of various data sources, which is time-consuming and increases the likelihood of human error. Furthermore, locating the exact location of an instrument based on these documents often requires the use of specialized computing tools and skills. These requirements further complicate the process, making location identification more difficult, especially in situations requiring rapid and accurate decision-making, such as quality inspections or maintenance activities. Given these limitations, there is a growing need for solutions that simplify the process of collecting coordinate information and other vehicle-related data for various processes. Such a method would significantly streamline the quality inspection or maintenance process and make coordinate information more accessible, especially to those with less technical knowledge. Summary of the Invention [Means for solving the problem]

[0003] The present disclosure provides, in one aspect, a method including the steps of generating a photographic record showing components within a vehicle, extracting primary coordinate data for the components by analyzing multidimensional modeling data, and integrating the primary coordinate data into the photographic record, wherein integrating the primary coordinate data into the photographic record includes defining a reference plane within the photographic record based on the primary coordinate data, identifying one or more sub-components within the defined reference plane, and generating secondary coordinate data for each of the one or more sub-components.

[0004] Another aspect of the present disclosure provides a system including one or more non-transitory computer-readable media containing, in any combination, computer program code that, when executed by operation of a computer system, performs operations according to one or more of the above methods, as well as one or more computer processors and one or more memories containing one or more programs that, when executed by the one or more computer processors, perform operations according to one or more of the above methods.

[0005] So that the above-cited features, briefly summarized above, may be understood in detail, a more particular description may be had by reference to exemplary embodiments, some of which are illustrated in the accompanying drawings. [Brief explanation of the drawings]

[0006] [Figure 1] FIG. 1 illustrates an exemplary coordinate data mapping system according to some aspects of the present disclosure. [Figure 2] FIG. 1 illustrates an example of a polygon coordinate interface (PCI) according to some aspects of the present disclosure. [Figure 3] FIG. 1 illustrates an example of an end user interface (EUI) according to some aspects of the present disclosure. [Figure 4]FIG. 1 illustrates an exemplary method for updating a photographic record with extracted spatial data, according to some aspects of the present disclosure. [Figure 5] FIG. 1 illustrates an exemplary method for integrating spatial data into photographic records and locating components, according to some aspects of the present disclosure. [Figure 6] 1 is a flow diagram illustrating an example method for coordinate data extraction and integration according to some aspects of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0007] The present disclosure provides a system that integrates coordinate information from technical data sources into photographic images of vehicle assemblies. The integration allows a user to interact with the image and receive precise coordinate data for any selected point. This system allows users to more efficiently and accurately locate and understand the location of specific components on a vehicle. Rapid availability of coordinate data is highly beneficial to operations that benefit from rapid identification of areas of interest within a vehicle, such as vehicle inspection and maintenance activities. The methods described in this disclosure improve the efficiency of these operations by making coordinate data more accessible to a wider range of users, regardless of technical expertise. Increased accessibility streamlines various operational activities and ensures accuracy and speed in identifying and assessing vehicle components.

[0008] FIG. 1 illustrates an exemplary coordinate data mapping system 100 .

[0009] The illustrated coordinate mapping system 100 includes a database 155, a back-end server 105, an administrator device 145, and an end-user device 150. The back-end server 105 includes two separate I / O interfaces: a polygon coordinate interface (PCI) 130 (as shown in FIG. 2 ) and an end-user interface (EUI) 135 (as shown in FIG. 3 ). These two interfaces are accessed via different devices: the PCI 130 is accessed by an administrator or system developer via device 145, and the EUI 135 is accessed by an end user via device 150. The back-end server is configured to generate a detailed visual representation (also referred to in some embodiments as a photographic record) of a vehicle's interior or exterior using 2D images and / or stitched images stored in the database 155. Upon generating the visual representation, the back-end server 105 retrieves technical documentation from the database 155 related to the assembly or component (e.g., MCP 235 in FIG. 2 ) shown in the representation. By analyzing the technical documentation, the backend server 105 determines coordinate data (e.g., 215 in FIG. 2) for each illustrated assembly or component (e.g., MCP 235 in FIG. 2) and then integrates the coordinate data into the photographic record. During integration, the backend server 105 may further define geometric planes (also referred to in some embodiments as polygonal planes) (e.g., plane 220-2 in FIG. 2) within the photographic record of each illustrated assembly or component (e.g., MCP 235 in FIG. 2). For each geometric plane, the backend server 105 may identify one or more subcomponents (e.g., switch 245 in FIG. 2) within the illustrated assembly or component and generate secondary coordinate data for each of these subcomponents.

[0010] In the illustrated example, database 155 communicates with backend server 105 via network interface 140 and stores data necessary for the server's operation. In some embodiments, database 155 may store various technical documents and data from which server 105 can extract coordinate information. In some embodiments, the coordinate information may refer to coordinates of data points that define the location of an object or component in three-dimensional (3D) space relative to a reference system (e.g., a vehicle's local coordinate system). In some embodiments, the coordinates of the data points may be represented by a different system, such as a Cartesian coordinate system or an air station coordinate system. The Cartesian coordinate system includes X, Y, and Z axes for specifying points in 3D space. The air station coordinate system, which specifically includes STA (station), WL (water line), and BL (buttock line) axes, is primarily designed for aircraft. In this system, STA represents the longitudinal position of a point along the length of the aircraft, WL represents the vertical position of a point along the height of the aircraft, and BL represents the lateral position of a point relative to the aircraft's centerline. In some embodiments, these technical documents may include, but are not limited to, multi-dimensional models (e.g., 3D models), detailed engineering drawings (providing dimensions and / or specifications of various parts within the vehicle), and installation instructions and diagrams. In some embodiments, the multi-dimensional model may include a complete 3D model of the entire vehicle for a comprehensive structure. In some embodiments, the multi-dimensional model may include detailed 3D models of specific areas or assemblies within or outside the vehicle, such as the flight control deck, passenger seats, landing gear, aircraft engines, etc.

[0011] In some embodiments, database 155 may store high-resolution images displaying specific assemblies or components within or outside the vehicle. These may include two-dimensional (2D) images for a more comprehensive view and / or stitched images (e.g., forming 180-degree or 360-degree panoramas). In some embodiments, database 155 may include additional reference material that provides context for multidimensional models and drawings, such as part numbers, material specifications, and manufacturing data. In some embodiments, the aforementioned data may be stored within local storage 120 of backend server 105, as opposed to a remote database 155. Such a configuration may enable faster data retrieval and processing and / or reduce dependency on network connectivity for some time-sensitive operations.

[0012] In some embodiments, backend server 105 may be configured to generate a detailed visual representation (also referred to in some embodiments as a photo record) of the interior or exterior of a vehicle using 2D images and / or stitched images (stored in database 155). The photo record may capture one or more assemblies or components within or outside the vehicle (e.g., a flight control deck with multiple control panels and / or display units) in 2D format from a particular viewpoint where a camera is positioned. The original photo record does not include specific coordinate information of the components it depicts, such as X, Y, Z coordinates and other position data.

[0013] In some embodiments, following generation of the photographic record, the backend server 105 may be configured to retrieve technical documentation from the database 155 related to the assembly or component (e.g., the flight control deck) depicted in the record. The technical documentation may include various materials, such as multidimensional models, engineering drawings, installation guidelines and diagrams, and other reference materials. The materials may depict the broader environment in which the assembly is located or may focus specifically on the assembly of interest. For example, assuming the photographic record depicts the flight control deck within an aircraft cockpit, the retrieved technical documentation may include various forms of 3D models, engineering drawings, and / or installation blueprints. Some of these materials may depict the entire cockpit, providing a complete visual representation of where the flight control deck is located and / or its relationship to other components within the cockpit (e.g., windows, control wheel, pilot seat). Others may be more focused on, such as a detailed 3D model specifically directed at the control panel itself. These noted materials may provide details of the flight control deck, such as the number of panels or display units contained within the flight control deck, the size, shape, and location of each panel, and / or the subcomponents (e.g., buttons, switches, or displays) within each panel and their corresponding dimensions.

[0014] In some embodiments, after retrieving technical documents from database 155, backend server 105 may analyze the documents to determine coordinate data for each illustrated assembly or component. For example, in the case of a three-dimensional model showing a flight control deck with multiple control panels or display units, server 105 may identify the coordinates (e.g., X, Y, Z or STA, BL, WL) of different control panels in the model. Additionally, server 105 may use image recognition technology to interpret the model, drawing, or guideline to understand the location of each control panel relative to other cockpit features (e.g., windows, control wheels, pilot seats) and extract dimensional data indicating the size and shape of each control panel in the flight control deck.

[0015] In some embodiments, the server 105 may integrate spatial data (e.g., coordinates of each control panel) into the photographic record. This may include defining a geometric plane (also referred to in some embodiments as a polygonal plane) within the photographic record for each assembly or component shown. If an assembly includes multiple sections (e.g., of a flight control deck with multiple control panels or display units), each with a different orientation, angle, or shape, a separate plane may be established for each section. Each geographic plane may be defined by a number of points (e.g., nine or fewer points) using coordinates obtained from technical documentation (e.g., a 3D model). In an embodiment where the photographic record shows a rectangular control panel (e.g., 230 in FIG. 2), the server 105 may extract the coordinates of four points defining the corners of the panel from the 3D model. These four points effectively outline the shape, size, and orientation of the panel in 3D space. The server 105 then uses the mapped points and their coordinates to draw a virtual plane (e.g., 220-1 in FIG. 2) within the record. The plane (eg, 220-1 in FIG. 2) acts as a reference for the position and orientation of the control panel (eg, 230 in FIG. 2) within the 2D image.

[0016] In some embodiments, the integration process may be performed manually by a developer via PCI 130. The developer may manually compare the 3D model or technical drawing with the photographic record to ensure that the extracted coordinate data corresponds to the assembly shown in the photographic record. The developer may then input the coordinates of the points into the system to construct a plane.

[0017] In some embodiments, once the plane is established, the server 105 may perform a verification process. This may include cross-referencing the established plane (e.g., 220-1 in FIG. 2 ) with the actual image of the assembly (e.g., 230 in FIG. 2 ) in the record. During this process, the server 105 may check for misalignment and / or discrepancies in size, shape, and / or location between the defined plane and the physical appearance of the assembly shown in the record. Verification seeks to confirm that the boundaries and proportions of the polygonal plane properly align with the edges, corners, and overall geometry of the assembly as shown in the photographic record. Inconsistencies identified during the verification process may be flagged for manual review via the PCI 130 to maintain accuracy of the spatial data mapping.

[0018] In some embodiments, after establishing the polygonal plane, the server 105 may proceed to determine the coordinates of other points within the plane. During this process, the server 105 may use the coordinates of the initial point (e.g., the four corner points of a rectangular plane) as a reference. The server 105 may then utilize geometric principles to map the interior area of ​​the plane and create a sub-coordinate system (with X and Y axes) that covers the entire surface area of ​​the assembly (e.g., 230 in FIG. 2). The sub-coordinate system may then identify points representing sub-components such as buttons, switches, and displays (e.g., 245 in FIG. 2) within the assembly and determine their relative positions with respect to the reference point. The 3D coordinates (e.g., X, Y, Z, or STA, WL, BL) of these sub-components may then be determined based on the reference coordinates of the initial point. In some embodiments, additional coordinate data may be overlaid on the 2D photographic record. The expanded record may then be made available to end users via the EUI 135. By utilizing the enhanced records, a user (e.g., an inspector or maintenance personnel) can quickly identify the precise location (e.g., 3D coordinates) of an assembly or its individual subcomponents with a single click.

[0019] In some embodiments, coordinate data (e.g., X, Y, Z, or STA, WL, BL) extracted from the model or drawing, as well as the initial photographic record and the extended record with spatial data overlay, may be stored in database 155 for further reference and use.

[0020] In some embodiments, the PCI 130 may be designed as a web-based platform accessible via a web browser 160 or part of an application 165 installed on the administrator device 145. Through the PCI 130, a developer (or administrator) may manually align the photographic record with an existing 3D model or technical drawing. The alignment process confirms that coordinates extracted from the model or drawing accurately correspond to the assembly or component shown in the record. In some embodiments, the developer may use the PCI 130 to input coordinates of identified reference points (e.g., the four corners of a rectangular control panel) into the system to construct a geometric plane representing the physical layout of the assembly in the photographic record. In some embodiments, the PCI 130 may provide tools for verifying the accuracy of the constructed plane and / or making any necessary adjustments.

[0021] In some embodiments, the EUI 135 may be configured as a web-based interface to facilitate access via a web browser 170. In some embodiments, the EUI 135 may be application-based, accessible via an application 175 within the end-user device 150. In some embodiments, the EUI may provide the user with access to augmented photographic records. These augmented records include overlay coordinates, geometric planes, and markers that pinpoint the precise location of each depicted assembly or component. In some embodiments, the user may interact with these displayed records via the EUI 135, such as by clicking on a component in the image to see detailed information about the component, such as its 3D coordinates. For example, if the record shows a control panel (e.g., 230 in FIG. 2) in the flight control deck, the user may click on a button, switch, display, or other component (e.g., 245 in FIG. 2) within the control panel. In response, the EUI 135 may reveal specific details about these components, such as their precise 3D coordinates relative to the aircraft (e.g., X, Y, Z, or STA, WL, BL), their dimensions, part number, and other relevant information.

[0022] FIG. 2 illustrates an example of a polygon coordinate interface (PCI) 200 according to some aspects of the present disclosure.

[0023] The illustrated diagram shows a web-based PCI 200. The web-based PCI includes two main windows: a photo record window 205 on the left and a coordinate entry window 210 on the right. The photo record window 205 displays a photo record (e.g., a 360-degree or 180-degree stitched image, a single two-dimensional image) of a vehicle (e.g., an aircraft). In some embodiments, the photo record shows a particular area within or outside the vehicle (e.g., the cockpit in an aircraft) and / or captures one or more assemblies therein. For example, the photo record shown in FIG. 2 shows a flight control deck including several control panels and / or display units, such as a control display unit (CDU) 240, a main instrument panel 230, and a model control panel (MCP) 235. As shown, each panel or display unit includes several subcomponents. For example, the main instrument panel 230 includes several screens that display important flight information, such as altitude, attitude, airspeed, vertical speed, and navigation information. The main instrument panel 230 also includes several keys or switches 245 in the center that interact with different display options or settings within the main instrument panel 230 .

[0024] For each control panel (e.g., 230, 235, 240), one or more geometric planes are constructed within the photographic record to define its boundaries and surface area. For the main instrument panel 230 and CDU 240, which have standard rectangular shapes, planes 220-1 and 220-3 can be constructed once the coordinates of their four corners (each corner has an X, Y, and Z location in 3D space) are determined. For the MCP 235, which has a more complex shape, polygonal plane 220-2 is defined. Unlike a rectangle, a polygon can have five or more sides, and therefore, five or more reference points can be used to define its shape. As shown, up to nine reference points 215 can be used in the coordinate entry window 210 to define a polygonal plane that matches the actual shape of the MCP in the record.

[0025] As discussed above, the coordinates of the reference points 215 can be determined by analyzing various technical documents and drawings (e.g., a 3D model of the cockpit, an engineering drawing of the flight control deck). In some embodiments, these coordinates may be automatically input into the system to construct the plane. In some embodiments, a developer may manually enter the 3D coordinates of each point using the coordinate input window 210. A manual process includes a more interactive approach in which a developer may select or click points (e.g., pixels) in the photographic record to create nodes 215 (also referred to as reference points in some embodiments). These nodes (or reference points) 215 may then be labeled with the corresponding coordinates (from the 3D model or engineering drawing) that represent that particular point. For example, to construct the plane 220-1 of the main instrument panel 230, which has a standard rectangular shape, a developer may start by selecting four corner points (or pixels) in the image. Based on the extracted data, the developer may label each of these points with a corresponding X coordinate 225-1, Y coordinate 225-2, and Z coordinate 225-3, or equivalent in an air station coordinate system (such as STA, WL, and BL coordinates). For components that do not have a standard shape, such as the MCP 235, the developer may select up to nine points (or pixels) to accurately outline the boundary of the component. Each of these points may then be assigned the appropriate X, Y, and Z coordinates (from the 3D model) to create a polygonal plane (e.g., 220-2) that precisely maps the irregular shape of the component in the record.

[0026] Although the illustrated PCI 200 shows the use of up to nine reference points 215 to define the polygonal plane, this is provided for conceptual clarity. In some embodiments, any number of reference points (not limited to nine) may be used to define the polygonal plane of a component or assembly.

[0027] Once the plane of each panel, such as plane 220-1 of main instrument panel 230, plane 220-2 of MCP 235, and plane 220-3 of CDU 240, is established, the relative positions of subcomponents (e.g., switches, buttons, keys, screens) within each panel can be identified. In some embodiments, a sub-coordinate system covering the entire surface of the plane is generated, allowing the position of each subcomponent to be determined relative to a reference point. The 3D coordinates (e.g., X, Y, Z, or STA, WL, BL) of each subcomponent, such as switch 245 on main instrument panel 230, may then be determined based on the known 3D coordinates of reference point 215. The coordinate data of each panel and its subcomponents may then be overlaid on a photographic record. This augmented record (with spatial data overlay) may then be provided to an end user via an EUI.

[0028] FIG. 3 illustrates an example of an end user interface (EUI) 300 according to some aspects of the present disclosure.

[0029] The illustrated diagram shows a web-based EUI 300 that includes a photo record window 305 as its primary feature. The photo record window 305 displays an expanded photo record showing a particular area or assembly within a vehicle, such as a flight control deck within an aircraft cockpit. In the lower right corner of the EUI 300 is a location display window 310. This window 310 dynamically shows the 3D coordinates (e.g., X, Y, Z, or STA, WL, BL) of any point or component within the record when selected. For example, to locate the main instrument panel 230 within an aircraft, a user simply clicks on any point (or pixel) of the panel within the photo record window 305, and the 3D coordinates (relative to the vehicle's local coordinate system) of the clicked point are displayed in the location display window 310. In some aspects, functionality may also extend to sub-components within each panel. For example, if a user wants to determine the location of a switch 245 within the main instrument panel 230, the user can click on the point (or pixel) representing the switch 245 within the photo record. The EUI responds by automatically displaying the 3D coordinates of the switch 245 in the location display window 310. The interactive capabilities of the EUI 300 may facilitate quick and efficient access to detailed location data. The EUI allows inspectors or maintenance personnel to quickly locate a specific component (e.g., the main instrument panel 230) or subcomponent (e.g., the switch 245) within the aircraft using its 3D coordinates. This approach is more efficient than traditional methods that require manual searching and referencing various technical documents to estimate the component's location. By providing precise, directly accessible location data, the EUI 300 simplifies the location estimation process and / or improves the efficiency and accuracy of maintenance and inspection procedures.

[0030] While the illustrated example shows PCI 200 and EUI 300 to be web-based, the example is provided for conceptual clarity only. In some aspects, PCI 200 or EUI 300 may be application-based, accessible by an application installed on an administrator device (e.g., 145 in FIG. 1), or may be in any other suitable format depending on particular requirements and user preferences.

[0031] 4 illustrates an example method 400 for updating a photo record with extracted spatial data according to some aspects of the present disclosure. In some aspects, method 400 may be performed by one or more computing systems or devices, such as backend server 105 as shown in FIG.

[0032] Method 400 begins at block 405, where a computing system (e.g., back-end server 105 of FIG. 1) generates a photographic record of a vehicle (e.g., an aircraft). The record serves as a visual representation of a particular area within or outside the vehicle (e.g., a cockpit within an aircraft) and / or may capture one or more assemblies within that area (e.g., a flight control deck with multiple control panels or display units). As discussed above, the photographic record may include two-dimensional images and / or stitched images. For 2D images, the process may involve using a strategically placed camera within the vehicle (e.g., an aircraft cockpit) and acquiring a single image directed toward the area of ​​interest (e.g., a flight control deck). For stitched images, the process may involve capturing multiple images from various angles / positions around the interior of the vehicle (e.g., the cockpit) and then stitching these individual images to create a 360-degree view of the entire space. The choice between 2D images and stitched images depends on the level of detail required and the spatial complexity of the area being recorded. In either case, the assemblies or instruments (such as cockpit control panels) shown in the resulting photographic records are presented in a flat format, and these images capture only the layout and appearance of these components without specific coordinate information.

[0033] In block 410, after the photographic record is generated, the computing system retrieves relevant technical documentation from a database. These documents may include, but are not limited to, 3D models, engineering drawings, and installation guidelines. These documents may provide detailed information about the vehicle components or assemblies (such as various control panels or display units in the flight control deck) as shown in the record. As discussed above, the materials may provide either a broad overview of the component's environment (e.g., a 3D model showing the entire cockpit) or specific details of the component of interest (e.g., a detailed 3D model specifically focused on the flight control deck).

[0034] In block 415, the computing system analyzes the technical documentation to extract coordinate data for components or assemblies depicted within the record. In some embodiments, the process may include using an image recognition algorithm to identify reference points for each depicted component. In embodiments where the record depicts a flight control deck having multiple control panels, such as a main instrument panel (e.g., 230 in FIG. 2), an MCP (e.g., 235 in FIG. 2), and a CDU (e.g., 240 in FIG. 2), the layout or appearance of each panel may be defined by specific reference points. For example, a main instrument panel having a rectangular shape may be defined by the coordinates of its four corner points. The computing system may analyze the technical documentation (e.g., 3D models, engineering drawings) to extract the 3D coordinates (e.g., X, Y, Z, or STA, WL, BL) of these reference points. In embodiments where different documents utilize various reference systems or units, the computing system may convert the extracted coordinates into a standardized format to ensure consistency and accuracy in the coordinate data integration process.

[0035] At block 420, the computing system integrates the extracted coordinate data into the photographic record. The process may include overlaying the coordinate data onto a 2D record showing the interior or exterior of the vehicle. In some embodiments, the integration converts the photographic record into an interactive map that facilitates accurate and precise identification of the location of each depicted component (e.g., main instrument panel 230 in FIG. 2) and / or its various subcomponents (e.g., switch 245 located on the main instrument panel as shown in FIG. 2). More details about the integration process are discussed below with reference to FIG. 5.

[0036] In block 425, the computer system updates the augmented photo record with the coordinate data overlay in the EUI (e.g., 300 in FIG. 3 ). Through the EUI, end users, such as inspectors or maintenance personnel, may interact with the record to obtain precise location data of components and / or subcomponents for efficient vehicle inspection and maintenance.

[0037] 5 illustrates an exemplary method 500 for integrating spatial data into photographic records and identifying component locations according to some aspects of the present disclosure. In some aspects, method 500 provides further details regarding spatial data integration, as shown in block 420 of FIG. 4. In some aspects, method 500 may be performed by one or more computing systems or devices, such as back-end server 105 as shown in FIG. 1.

[0038] In block 505, a computing system (e.g., back-end server 105 of FIG. 1) uses the extracted coordinate data to construct a geometric plane (e.g., a rectangle or polygon) within the photographic recording. For components having a rectangular shape (such as main instrument panel 230 shown in FIG. 2), the coordinates of four corner points may be used. For components having a non-standard shape (e.g., MCP 235 of FIG. 2), this process may involve using five or more points (e.g., up to nine) to precisely delineate a plane that precisely aligns with the shape of the component. The computing system may connect these reference points to form the edges of the plane (e.g., a rectangle or polygon). Once the plane is defined, the computing system overlays the plane on the photographic recording to align the plane within the visual representation of the component within the recording. In some embodiments, the process of defining and overlaying planes within the recording may be an automated process, in which the computing system automatically calculates and constructs geometric planes based on the extracted coordinate data and aligns them with corresponding components in the 2D space of the recording. In some embodiments, the process may be performed manually by a developer via the PCI (e.g., 200 in FIG. 2). In such a configuration, the developer may manually enter the coordinates of the reference points and overlay the plane on the photographic record.

[0039] In block 510, the computing system performs a validation process to check for discrepancies between the constructed plane and the actual visual representation of the component or assembly in the record. The validation process enables the computing system to ensure that the size, shape, and position of the overlaid plane accurately matches the corresponding component as it appears in the 2D image. If discrepancies are detected, the computing system may trigger a warning and / or flag these areas for further review. In some embodiments, the developer may review and fine-tune the plane via PCI (e.g., 200 of FIG. 2).

[0040] In block 515, the computing system identifies additional items or subcomponents (e.g., switches, buttons) within the established plane. While technical documentation such as 3D models provides comprehensive information, they may not always include detailed coordinate data for each individual switch or button (e.g., switch 245 in FIG. 2) within a control panel. The computing system may use image recognition algorithms to detect these subcomponents and determine their relative positions within the plane. In some embodiments, the relative positions may be determined by defining a subcoordinate system within each plane (having an X-axis and a Y-axis). The subcoordinate system may serve as a reference for mapping the positions of the subcomponents. For example, within a plane (e.g., 220-1 in FIG. 2) representing the main instrument panel (e.g., 230 in FIG. 2), a switch (e.g., 245 in FIG. 2) is assigned X and Y coordinates relative to the plane's origin, which is one of the plane's reference points.

[0041] In block 520, following identification of the sub-components and their relative positions within the defined plane, the computing system utilizes the known 3D coordinates of the reference points of the plane to calculate the full 3D coordinates of each sub-component (which may not be explicitly detailed in existing technical documentation). The full 3D coordinate data of each sub-component may then be integrated into the photographic record, thereby enabling an end user via the EUI (e.g., 300 in FIG. 3) to quickly and accurately locate the sub-components within the vehicle, even if the data is not explicitly disclosed in existing technical documentation.

[0042] FIG. 6 is a flow diagram illustrating an example method 600 for coordinate data extraction and integration according to some aspects of the present disclosure.

[0043] In block 605, a computing system (eg, backend server 105 of FIG. 1) generates a photographic record showing a component (eg, main instrument panel 230 of FIG. 2).

[0044] At block 610, the computing system extracts primary coordinate data for the component by analyzing one or more technical documents (as illustrated by blocks 410 and 415 of FIG. 4 ). In some aspects, the one or more technical documents may include at least one of one or more three-dimensional (3D) models of the vehicle, one or more detailed 3D models focused on the component on the vehicle, one or more engineering drawings associated with the component or vehicle, or one or more installation documents associated with the component or vehicle.

[0045] In block 615, the computing system integrates the primary coordinate data into the photographic record (as indicated by block 420 of FIG. 4). The integration includes three steps, as discussed below with reference to blocks 620, 625, and 630. In some aspects, the integration may be performed via a polygon coordinate interface (PCI).

[0046] At block 620, the computing system defines a reference plane (e.g., 220-1 in FIG. 2) within the photographic record based on the primary coordinate data (as indicated by block 505 in FIG. 5). In some embodiments, the process of defining the reference plane within the photographic record may include determining a set of points (e.g., 215 in FIG. 2) indicative of components within the photographic record and labeling the set of points based on the primary coordinate data. In some embodiments, the set of points indicative of components within the photographic record may be determined using an image recognition algorithm.

[0047] In block 625, the computing system identifies one or more sub-components (eg, switch 245 in FIG. 2) within the defined plane (eg, 220-1 in FIG. 2).

[0048] At block 630, the computing system generates secondary coordinate data for each of the one or more subcomponents (as illustrated by block 520 of FIG. 5). In some aspects, the secondary coordinate data may include coordinates of each of the one or more subcomponents.

[0049] In some embodiments, the integration further includes performing a verification process to verify that the defined plane is correctly aligned with the visual features of the components indicated by the photographic record (as indicated by block 510 of FIG. 5 ).

[0050] In some aspects, in response to integrating the primary coordinate data into the photographic record, the computing system may further provide the photographic record augmented with the primary and secondary coordinate data for interaction with a user.

[0051] Various embodiments are referenced in this disclosure. However, it should be understood that the disclosure is not limited to the specific described embodiments. Instead, any combination of the following features and elements, whether associated with different embodiments, is contemplated to implement and practice the teachings provided herein. Furthermore, when elements of an embodiment are described in the form of "at least one of A and B," it is understood that embodiments including only element A, including only element B, and including elements A and B are respectively contemplated. Furthermore, while some embodiments may achieve other possible solutions and / or advantages over the prior art, whether or not a particular advantage is achieved by a given embodiment does not limit the disclosure. Accordingly, the embodiments, features, aspects, and advantages disclosed herein are merely exemplary and should not be considered elements or limitations of the appended claims unless expressly recited in the claims. Similarly, references to "the present invention" should not be construed as a generalization of the inventive subject matter disclosed herein, and should not be considered elements or limitations of the appended claims unless expressly recited in the claims.

[0052] As will be appreciated by those skilled in the art, aspects described herein may be embodied as a system, method, or computer program product. Accordingly, aspects may take the form of entirely hardware aspects, entirely software aspects (including firmware, resident software, microcode, etc.), or aspects combining software and hardware aspects, all of which may be referred to generally herein as "circuits," "modules," or "systems." Furthermore, aspects described herein may take the form of a computer program product embodied in one or more computer-readable storage medium(s) having computer-readable program code embodied therein.

[0053] The program code embodied on the computer readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, fiber optic cable, RF, etc., or any suitable combination of the foregoing.

[0054] Computer program code for carrying out operations for aspects of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and the like, and conventional procedural programming languages ​​such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partially on the user's computer as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter described scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or a connection may be made to an external computer (e.g., through the Internet using an Internet Service Provider).

[0055] Aspects of the present disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to aspects of the present disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to produce a machine such that the instructions, executed by the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the blocks of the flowchart illustrations and / or block diagrams.

[0056] These computer program instructions may be stored in a computer-readable medium that can direct a computer, other programmable data processing apparatus, or other device to function in a particular manner, such that the instructions stored in the computer-readable medium produce an article of manufacture that includes instructions that implement the functions / acts specified in the flowchart and / or block diagram blocks.

[0057] The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device such that a series of operational steps executed on the computer, other programmable data processing apparatus, or other device produce a computer-implemented process such that the instructions executing on the computer, other programmable data processing apparatus, or other device provide a process for implementing the functions / acts specified in the flowchart and / or block diagram blocks.

[0058] The flowchart diagrams and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various aspects of the present disclosure. In this regard, each block in a flowchart diagram or block diagram may represent a module, segment, or portion of code, including one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks may occur in an order different from the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially simultaneously, or the blocks may be executed in the reverse order or out of order, depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or a combination of dedicated hardware and computer instructions.

[0059] While the foregoing is directed to aspects of the present disclosure, other and further aspects of the disclosure may be devised without departing from the basic scope thereof, which scope is determined by the following appended claims. [Explanation of symbols]

[0060] 100 Coordinate data mapping system, coordinate mapping system 105 Backend Server 115 memory 120 Local Storage 125 Interconnect Bus 140 network interface 145 Administrator Devices 150 end-user devices 155 Remote Databases 160 web browsers 165 Applications 170 Web Browser 175 Applications 205 Photo Recording Window 210 Coordinate input window 215 Reference Points, Nodes 230 Main Instrument Panel 245 Switch 305 Photo Recording Window 310 Position Display Window

Claims

1. 1. A method comprising: generating a photographic record showing the component (235); extracting primary coordinate data (215) of said components by analyzing one or more technical documents; Integrating the primary coordinate data (215) into the photographic record, defining a reference plane (220) within said photographic record based on said primary coordinate data; Identifying one or more sub-components (245) within the defined reference plane; generating secondary coordinate data for each of said one or more sub-components; Including steps and A method comprising:

2. 2. The method of claim 1, wherein the step of integrating the primary coordinate data into the photographic record further comprises performing a verification process to verify that the defined reference plane (220) is correctly aligned with visual features of the component (235) indicated by the photographic record.

3. defining the reference plane within the photographic record, determining a set of points indicative of said component within said photographic record; and labeling the set of points based on the primary coordinate data.

4. The method of claim 3 , wherein the set of points representing the component in the photographic record is determined using an image recognition algorithm.

5. 2. The method of claim 1, wherein the one or more technical documents include at least one of one or more three-dimensional (3D) models of a vehicle, one or more detailed 3D models focused on the component on the vehicle, one or more engineering drawings associated with the component or the vehicle, or one or more installation documents associated with the component or the vehicle.

6. The method of claim 1 , wherein the secondary coordinate data includes coordinates of each of the one or more subcomponents (245).

7. The method of claim 1 , further comprising the step of providing, in response to integrating the primary coordinate data into the photographic record, the photographic record augmented with the primary and secondary coordinate data for interaction with a user.

8. A system (105), Polygon Coordinate Interface (PCI) (130); End User Interface (EUI) (135); one or more computer processors (110); one or more memories (115) collectively containing one or more programs that perform operations when executed by the one or more computer processors, the operations including: generating a photographic record showing the component (235); extracting primary coordinate data (215) of said components by analyzing one or more technical documents; Integrating the primary coordinate data (215) into the photographic record, defining a reference plane (220) within said photographic record based on said primary coordinate data; Identifying one or more sub-components (245) within the defined reference plane; generating secondary coordinate data for each of said one or more sub-components; Including, integrating and one or more memories, including A system comprising:

9. 9. The system of claim 8, wherein the one or more programs, when executed on any combination of the one or more computer processors (110), perform the operations to incorporate the primary coordinate data into the photographic record, and the operations further include performing a verification process to verify that the defined reference plane (220) is correctly aligned with visual features of the component (235) indicated by the photographic record.

10. the one or more programs, when executed on any combination of the one or more computer processors (110), perform the operations to define the reference plane within the photographic record, the operations comprising: determining a set of points indicative of said component within said photographic record; labeling the set of points based on the primary coordinate data; The system of claim 8, comprising:

11. The system of claim 10 , wherein the set of points representing the component in the photographic record is determined using an image recognition algorithm.

12. 9. The system of claim 8, wherein the one or more technical documents include at least one of one or more three-dimensional (3D) models of a vehicle, one or more detailed 3D models focused on the component on the vehicle, one or more engineering drawings associated with the component or the vehicle, or one or more installation documents associated with the component or the vehicle.

13. The system of claim 8 , wherein the secondary coordinate data includes coordinates of each of the one or more subcomponents (245).

14. 9. The system of claim 8, wherein the one or more programs, when executed on any combination of the one or more computer processors, perform the operations, the operations further comprising, in response to integrating the primary coordinate data into the photographic record, providing the photographic record augmented with the primary and secondary coordinate data for interaction with a user.

15. One or more non-transitory computer-readable media containing computer program code, in any combination, that, when executed by a computer system (105), performs operations, said operations including: generating a photographic record showing the component (235); extracting primary coordinate data (215) of said components by analyzing one or more technical documents; Integrating the primary coordinate data (215) into the photographic record, defining a reference plane (220) within said photographic record based on said primary coordinate data; Identifying one or more sub-components (245) within the defined reference plane; generating secondary coordinate data for each of said one or more sub-components; Including, integrating and [0023] 1. One or more non-transitory computer-readable media,

16. 16. The one or more non-transitory computer-readable media of claim 15, wherein the computer program code, when executed by a computer system, performs the operations to incorporate the primary coordinate data into the photographic record, and the operations further include performing a verification process to verify that the defined reference plane (220) is correctly aligned with visual features of the component (235) indicated by the photographic record.

17. The computer program code, when executed by a computer system, performs the operations to define the reference plane within the photographic record, the operations comprising: determining a set of points indicative of said component within said photographic record; labeling the set of points based on the primary coordinate data; 16. The one or more non-transitory computer-readable media of claim 15, comprising:

18. 20. The one or more non-transitory computer-readable media of claim 17, wherein the set of points indicative of the component within the photographic record is determined using an image recognition algorithm.

19. 17. The one or more non-transitory computer-readable media of claim 16, wherein the one or more technical documents include at least one of one or more three-dimensional (3D) models of a vehicle, one or more detailed 3D models focused on the components on the vehicle, one or more engineering drawings associated with the components or the vehicle, or one or more installation documents associated with the components or the vehicle.

20. 17. The one or more non-transitory computer-readable media of claim 16, wherein the computer program code, when executed by a computer system, performs the operations, the operations further comprising, in response to integrating the primary coordinate data into the photographic record, providing the photographic record augmented with the primary and secondary coordinate data for interaction with a user.

Images