Apparatus and method for monitoring a trained machine learning process associated with positioning determination

Abstract

[0003] A method, system, and apparatus for monitoring or configuring one or more user equipment (UE) devices. For example, a computing device may send a report request regarding positioning data to the user equipment. In addition, the computing device may receive report data indicating a comparison between first positioning data generated from a first process and second positioning data generated from a second process. Furthermore, the computing device may generate and send to the user equipment instructions, based on the report data, that cause the user equipment to perform the first process.

Description

[Technical Field]

[0001] The disclosed embodiments relate generally to monitoring a trained machine learning process. [Background technology]

[0002] Wireless communication systems can provide various telecommunication services, including, for example, audio, video, data, messaging, and network access, among others. For example, wireless communication systems may enable communication between various devices, such as Internet of Things (IoT) devices. These wireless communication systems may be based on various technologies, such as code division multiple access (CDMA) systems, time division multiple access (TDMA) systems, frequency-division multiple access (FDMA) systems, orthogonal frequency-division multiple access (OFDMA) systems, single-carrier frequency-division multiple access (SC-FDMA) systems, time division synchronous code division multiple access (TDSCDMA) systems, Long Term Evolution (LTE) systems, WiMax systems, and Evolved High Speed ​​Packet Access (HSPA+) systems. These and other wireless communication systems may conform to standards such as the third generation (3G) of broadband cellular network technology, the fourth generation (4G) of broadband cellular network technology, and the more recent fifth generation (5G) of broadband cellular network technology (also known as New Radio (NR)).

[0003] A wireless communication system may include several base stations (BSs) and several user equipments (UEs). In some examples, the BSs may enable wireless communication for several UEs. In addition, the wireless communication system may also provide location services. For example, the wireless communication system may include a location management function (LMF) that can provide location services to several UEs. Summary of the Invention

[0004] According to one aspect, an apparatus may include a non-transitory machine-readable storage medium storing instructions and at least one processor coupled to the non-transitory machine-readable storage medium. The at least one processor may be configured to send a report request for positioning data to a user equipment. In some examples, the report request may cause the user equipment to operate in a first mode. In some cases, the operation may include performing a first process to generate first positioning data, performing a second process to generate second positioning data, and generating report data indicative of a comparison between the first positioning data and the second positioning data. Additionally, the at least one processor may be configured to receive the report data from the user equipment. Furthermore, the at least one processor may be configured to generate and send to the user equipment instructions, based on the report data, that cause the user equipment to perform at least one of the first process or the second process.

[0005] According to another aspect of the present invention, instructions on a non-transitory machine-readable storage medium, when executed by at least one processor of a location server, cause the at least one processor to perform operations including sending a report request for positioning data to a user equipment. In some examples, the report request may cause the user equipment to operate in a first mode. In some cases, operation in the first mode may include performing a first process to generate first positioning data, performing a second process to generate second positioning data, and generating report data indicating a comparison between the first positioning data and the second positioning data. Additionally, the operations may include receiving the report data from the user equipment. Further, the operations may include generating and sending to the user equipment instructions that cause the user equipment to perform at least one of the first process and / or the second process based on the report data.

[0006] According to another aspect, a computer-implemented method includes sending a report request for positioning data to a user equipment. In some examples, the report request may cause the user equipment to operate in a first mode. In some cases, operation in the first mode may include performing a first process to generate first positioning data, performing a second process to generate second positioning data, and generating report data indicative of a comparison between the first positioning data and the second positioning data. Additionally, the computer-implemented method may include receiving the report data from the user equipment. Furthermore, the computer-implemented method may include generating and sending to the user equipment instructions, based on the report data, that cause the user equipment to perform at least one of the first process and / or the second process.

[0007] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention as claimed. Moreover, the accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate aspects of the disclosure and, together with the description, serve to explain the principles of the disclosed embodiments as set forth in the appended claims. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a block diagram of an exemplary wireless communication system, in accordance with some exemplary embodiments. [Figure 2] 1 is a block diagram illustrating a portion of an exemplary wireless communication system, in accordance with some exemplary embodiments. [Figure 3] 1 is a block diagram illustrating a portion of an exemplary wireless communication system, in accordance with some exemplary embodiments. [Figure 4] 1 is a block diagram illustrating a portion of an exemplary wireless communication system, in accordance with some exemplary embodiments. [Figure 5] 1 is a block diagram illustrating a portion of an exemplary wireless communication system, in accordance with some exemplary embodiments. [Figure 6] 1 is a block diagram illustrating a portion of an exemplary wireless communication system, in accordance with some exemplary embodiments. [Figure 7] 7 is a flowchart of an example process 700 for determining performance of a process deployed by a UE 104. [Figure 8] 8 is a flowchart of an example process 800 for generating reporting data, according to an example embodiment.

[0009] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION OF THE INVENTION

[0010] While the features, methods, devices, and systems described herein may be embodied in a variety of forms, some exemplary and non-limiting embodiments are shown in the drawings and described below. Some of the components described in this disclosure are optional, and some implementations may include additional, different, or fewer components than those explicitly described in this disclosure.

[0011] Embodiments described herein are directed to a wireless communications system including a computing device or system and several user equipments (UEs). Such embodiments may enable the computing device or system to monitor the performance of a particular process implemented in one or more of several UEs. In addition, while the computing device or system is monitoring the performance of a particular process implemented in each of the one or more UEs, each of the one or more UEs may be configured to implement a first mode, i.e., a reporting mode. When the corresponding UE is in the first mode, the UE may generate first data, such as first measurement data, associated with the particular process and second data, such as second measurement data, associated with a legacy process. Moreover, the computing device or system may determine the performance of the particular process based in part on the first data and the second data. Furthermore, the computing device or system may modify or change the configuration of one or more UEs based on the performance of the particular process, for example, causing one or more UEs to operate in the second mode instead of the first mode. In some cases, while the corresponding UE is operating in the second mode, the corresponding UE may perform another process to generate third data, such as third measurement data, in which case the third data may be utilized by a computing device or system to provide location services to the corresponding UE.

[0012] A. Exemplary Wireless Communication System 1 shows a block diagram of an exemplary wireless communications system 100, such as a 5G wireless communications system, including, among other things, one or more computing systems, such as location management function (LMF) computing system 102A, LMF computing system 102B, LMF computing system 102C, LMF computing system 102D, LMF computing system 102E, and LMF computing system 102F, at least one base station (BS) 103, and one or more user equipment (UE) 104, such as UE 104A, UE 104B, UE 104C, UE 104D, UE 104E, and UE 104F. Each of the one or more computing systems, such as LMF computing system 102, and the one or more UEs 104 may each be operatively connected to, and interconnected across, one or more communications networks. Wireless communication system 100 may include additional components, such as access and mobility management functions (AMFs), session management functions (SMFs), relay stations, and any other suitable components, which are not shown for simplicity. Additionally, although wireless communication system 100 may show only one BS 103, six LMF computing systems 102, and six UEs 104, wireless communication system 100 may include any number of LMF computing systems 102, BSs 103, and UEs 104.

[0013] LMF computing systems 102, such as LMF computing system 102A, LMF computing system 102B, LMF computing system 102C, LMF computing system 102D, LMF computing system 102E, and LMF computing system 102F, may each represent a computing system including one or more servers, such as server 202, and one or more tangible, non-transitory memory devices that store executable code, application engines, or application modules. Each of the one or more servers may include one or more processors that may be configured to execute portions of stored code, application engines or modules, or application programs to perform operations consistent with the disclosed exemplary embodiments. For example, with reference to FIG. 2, one or more servers of LMF computing system 102A may include server 202 having one or more processors configured to execute stored code, application engines or modules, or portions of application programs maintained in one or more tangible, non-transitory memories.

[0014] In some cases, LMF computing system 102 may correspond to a discrete computing system, while in other cases, LMF computing system 102 may correspond to a distributed computing system having multiple computing components distributed across a suitable computing network. Additionally, LMF computing system 102 may also include one or more communication interfaces, such as one or more wireless transceivers, coupled to one or more processors for facilitating wired or wireless Internet communications with other computing systems across the communications network, and devices operating within wireless communication system 100 (not shown in FIG. 1), such as additional components of wireless communication system 100, e.g., an Access and Mobility Management Function (AMF), a Session Management Function (SMF), a relay station, and any other suitable components.

[0015] In some examples, the LMF computing system 102 may be associated with a wireless communication service provider and may be operated by one or more operators of the wireless communication service provider. Additionally, the LMF computing system 102 may be configured to provide supporting location services to each of one or more UEs 104. For example, referring to FIG. 1 , the LMF computing system 102A may receive measurement data from the UE 104A for each beam transmitted by the BS 103 and detected by the UE 104A. In some cases, each element of the measurement data may include measurement information for the detected beam determined by the corresponding UE 104. Additionally, the measurement information may be associated with specific data or attributes (e.g., assistance data, positioning frequency layer (PFL) ID(s), positioning reference signal (PRS) resource ID, TRP, PRS resource set ID, timestamp, etc.). Examples of measurement information that may be included in the measurement data include reference signal time difference (RSTD), received signal received power (RSRP), reference signal received power path (RSRPP), time difference between received and transmitted measurements (Rx-Tx), and information characterizing a location estimate of a corresponding UE 104, such as UE 104A. The LMF computing system 102 may provide location services to a corresponding UE 104, such as generating and transmitting assistance data to UE 104A, based on the measurement data of each of one or more UEs 104. The assistance data may include, for example, a reference time, a reference location, an ionospheric model, Earth orientation parameters, time offsets, differential corrections, ephemeris and clock models, health status, data bit assistance, acquisition assistance, an almanac, a UTC model, and carrier phase data.In some cases, one or more UEs 104 may request location services, such as assistance data, from the LMF computing system 102. In such cases, the LMF computing system 102 may provide the location services to each of the one or more UEs 104 that requested the location services.

[0016] In another example, the LMF computing system 102 may monitor the performance of a new process performed by one or more of the UEs 104. Additionally, the LMF computing system 102 may monitor the performance of the new process based on measurement data associated with the new process and measurement data associated with the legacy process. In such an example, one or more UEs 104 may each perform the new process and the legacy process and generate measurement data associated with the new process and the legacy process. In some cases, the LMF computing system 102 determines the performance of the new process by comparing the measurement data of the new process with the measurement data of the legacy process. Additionally, the LMF computing system 102 may determine a difference between the measurement data of the new process and the measurement data of the legacy process and determine whether the difference exceeds a difference threshold or is below a quality / standard threshold. In instances where the LMF computing system 102 determines that the difference between the measurement data of the new process and the measurement data of the legacy process exceeds a quality or difference threshold (e.g., falls below a predetermined standard), the LMF computing system 102 may modify or change the configuration of one or more UEs 104. In some cases, the new process may be associated with a trained machine learning process. In other cases, the legacy process may be associated with a trained machine learning process.

[0017] 2, to facilitate performance of one or more of these exemplary processes, an LMF computing system 102, such as LMF computing system 102A, may maintain a data repository, such as data repository 204, including, but not limited to, a UE data store, such as UE data store 206, in one or more tangible, non-transitory memories. As shown in FIG. 2, a UE data store, such as UE data store 206, may store UE data sets for each of one or more UEs 104 in communication with the LMF computing system 102. As described herein, the UE data sets for each of the UEs 104, such as UE data 206A associated with UE 104A, may include data identifying the corresponding UE 104 (e.g., a corresponding serial number or identification number), data identifying one or more processes that may be performed by the corresponding UE 104 (e.g., a new first trained machine learning process and a second legacy trained machine learning process), data including parameters for each of the one or more processes, such as model parameters, and data indicating a performance status of the one or more processes. In some cases, examples of performance statuses of one or more processes include a performance status indicating that the performance of a particular process is below a predetermined standard and a performance status indicating that the performance of a particular trained machine learning process is below a predetermined standard.

[0018] Additionally, to facilitate performance of any of the example processes described herein, an LMF computing system 102, such as LMF computing system 102A, may include one or more servers, such as server 202, that may also maintain an application repository, such as application repository 208, in one or more tangible, non-transitory memories. By way of example, application repository 208 (or any application repository of any LMF computing system 102) may maintain, among other things, a UE engine 208A. UE engine 208A may initiate monitoring of the performance of one or more processes performed by each of one or more UEs 104 by generating and sending (e.g., broadcasting) a report request to each of one or more UEs 104. In some examples, the report request may include parameter data identifying one or more monitoring parameters. In some cases, each of one or more UEs 104 may generate measurement data from a particular process, such as a new trained machine learning process, and from a legacy process that may be equivalent for purposes of determining the performance of the particular process, according to the one or more monitoring parameters. For example, the measurement data generated from the particular process and the legacy process may each include the same types of data or attributes and may have matching or similar timestamps. Further, each of the one or more UEs 104 may generate report data indicative of the performance of the particular process based on the comparable measurement data of the particular process and the measurement data of the legacy process.

[0019] In some examples, one or more monitoring parameters of the parameter data may include timing parameters. The timing parameters may indicate a time period during which the corresponding UE 104 may perform the monitored process and the legacy process, such that the generated corresponding measurement data may have elements with timestamps that match or fall within a predetermined time threshold or margin. In such examples, the monitored process, such as a new trained machine learning process, and the legacy process, such as a trained legacy machine learning process, may each have different measurement period configurations and may detect and determine measurements at different time intervals (e.g., RAN 4 requirements). Thus, the timing parameters may indicate an extended measurement period for both the monitored process and the legacy process (e.g., extending RAN 4 requirements). In this way, one or more elements of the measurement data of the monitored process and one or more elements of the measurement data of the legacy process may have timestamps that match or fall within a predetermined time threshold or margin. In other examples, the one or more monitoring parameters of the parameter data may include model or process parameters. The model or process parameters may identify a particular process that the LMF computing system 102 may monitor or that the model UE 104 may perform.

[0020] In various examples, one or more monitoring parameters of the parameter data may include a resource parameter. The resource parameter may identify one or more resources, attributes, or data that the corresponding UE 104 may measure from one or more detected beams transmitted from the BS 103. In some cases, such resources, attributes, or data may be associated with assistance data, PFL ID(s), PRS resource ID, TRP, and PRS resource set ID. Furthermore, the UE 104 may perform a monitored process and a legacy process based on the resource parameter, each of which generates measurement data including measurements or measurement information of the same resources, attributes, or data. For example, the UE 104 may perform a monitored process together with a legacy process based on the resource parameter, each of which generates measurement data including data identifying an RSTD. In other cases, the resource parameter may identify a subset of resources, attributes, or data, e.g., a subset of TRP resources, that the corresponding UE 104 may measure from one or more detected beams transmitted from the BS 103. In such a case, the UE 104 may implement at least two processes, including a monitored process and a legacy process, based on the resource parameters, each process generating measurement data (e.g., measurements, measurement information) for the same subset of resources, attributes, or data.

[0021] In various cases, the resource parameter may identify a first subset of resources, attributes, or data and a second set of resources, attributes, or data that the corresponding UE 104 may measure from one or more beams transmitted from the BS 103. Additionally, the resource parameter may indicate which of a process, such as a measured process or a legacy process, generates measurement data associated with which subset of resources (e.g., the first subset of resources or the second subset of resources). For example, the resource parameter may indicate that the first subset of resources is associated with a legacy process, where the first subset of resources includes TRPs, such as a reference TRP, a line-of-sight (LOS) heavy TRP, a non-line-of-sight (NLOS) TRP, a serving TRP, or any combination thereof. Additionally, the resource parameter may indicate that the second subset of resources is associated with the measured process. For example, the second set of resources may include resources, attributes, or data associated with assistance data, a PFL ID, a PRS resource ID, a TRP, a PRS resource set ID, a PRS resource, or any combination thereof. Moreover, based on the resource parameters, the UE 104 may perform a legacy process to generate measurement data for the first subset of resources, attributes, or data and perform a monitored process to generate measurement data for the second subset of resources, attributes, or data.

[0022] In some examples, the parameter data of the report request may be associated with a particular positioning method (e.g., UE-assisted or UE-based). In such examples, the measurement data may then be associated with the positioning method with which the parameter data is associated. For example, the report request may include parameter data associated with a multi-RTT positioning method. In such examples, the corresponding UE 104 may implement a monitored first process and a second legacy process, both of which generate measurement data associated with the multi-RTT positioning method. For example, the measurement data may include data identifying the UE Rx-Tx, RSRP, and / or RSRPP for each of the first process and the second legacy process. In yet another example, the report request may include parameter data associated with a DL-AoD positioning method. In such examples, the corresponding UE 104 may implement a monitored process and a second legacy process, both of which generate measurement data associated with the DL-AoD positioning method. For example, the measurement data may include data identifying the RSRP and / or RSRPP. In another example, the report request may include parameter data associated with a UE-based positioning method. In such an example, the corresponding UE 104 may perform a monitored process and a second legacy process, both of which generate measurement data associated with the positioning method. For example, the measurement data may include data that identifies and characterizes a location estimate for the UE.

[0023] Additionally, as shown in FIG. 2 , the UE engine 208A may modify or change the configuration of one or more UEs 104 based on the measurement data of the monitored process and the measurement data of the legacy process. For example, the UE engine 208A may receive report data generated by each of the one or more UEs 104. The report data may indicate the performance of the monitored process. In some cases, the report data may indicate whether the performance of the process exceeds a predetermined standard or falls below a quality threshold. In an example where the UE engine 208A determines, based on the report data, that the performance of the monitored process falls below a quality / standard threshold, the UE engine 208A may modify or change the configuration of the corresponding one or more UEs 104. For example, the report data may indicate that a difference between the measurement data of the new process and the measurement data of the legacy process exceeds a difference threshold. In an example where the UE engine 208A determines, based on the report data, that a difference between the measurement data of the new process and the measurement data of the legacy process exceeds a difference threshold, the UE engine 208A may modify or change the configuration of one or more UEs 104.

[0024] In some cases, the report data may include raw measurement data generated by a particular process being monitored and raw measurement data generated by another process being used to determine the performance of the particular process. In such cases, an LMF computing system 102, such as LMF computing system 102A, may process the raw measurement data of the second measurement data to determine the accuracy of the first measurement data and whether the performance of the particular process being monitored is being monitored. For example, the LMF computing system 102 may obtain report data for a UE 104, such as UE 104A. The report data may include measurement data of a first trained machine learning process applied by the UE 104 to detected beams transmitted from the BS 103. The measurement data may include data characterizing and identifying a location estimate for the UE 104. Additionally, the report data may include raw measurement data of a second legacy process applied by the UE 104 to detected beams transmitted from the BS 103. The LMF computing system 102 may determine a location estimate for the UE derived from the raw measurement data of the second legacy process based on the raw measurement data of the second legacy process. Further, the LMF computing system 102 may compare the location estimate of the first trained machine learning process with the location estimate determined from the second legacy process to determine the accuracy and performance of the first trained machine learning process. The LMF computing system 102 may modify or alter the configuration of the UE 104 based on such determination. For example, the LMF computing system 102 may determine a difference between a value associated with the location estimate of the first trained machine learning process and a value associated with the location estimate determined from the second legacy process. Additionally, the LMF computing system 102 may determine whether the determined difference exceeds a difference threshold.In response to the LMF computing system 102 determining that the determined difference exceeds the difference threshold, the LMF computing system 102 may modify or change the configuration of the UE 104 (e.g., operate in a mode that utilizes the second, legacy, or another process instead of the first trained machine learning process to generate the measurement data). Otherwise, in examples where the determined difference is less than or equal to the difference threshold, the LMF computing system 102 may allow the UE 104 to continue performing the first trained machine learning process to generate the measurement data.

[0025] Referring to FIG. 1 , BS 103, which may also be referred to as a Node B, gNB, transmit receive point (TRP), access point (AP), etc., may provide communication coverage for a particular geographic area, such as geographic area 101. For example, geographic area 101 may correspond to a macrocell, a picocell, a femtocell, or any other type of cell. To provide coverage, BS 103 may transmit one or more beams that cover at least a portion of geographic area 101. Each beam may include one or more carriers operating within a frequency spectrum. For example, BS 103 may transmit data, such as a PRS, in a downlink transmission to one or more UEs 104 using one or more carriers associated with each beam.

[0026] Additionally, each of the UEs 104, such as UE 104A, UE 104B, UE 104C, UE 104D, UE 104E, and UE 104F, may each detect one or more beams transmitted from the BS 103. Further, each of the UEs 104 may deploy one or more processes for the detected one or more beams to generate corresponding measurement data. Referring to FIG. 2, to facilitate performance of one or more of these example processes, each of the UEs 104, such as UE 104A, may include a computing device having one or more tangible, non-transitory memories, such as memory 212, that store data and / or software instructions, and one or more processors, such as processor 214, configured to execute the software instructions. The one or more tangible non-transitory memories may, in some aspects, store application programs, application engines or modules, and other elements of code executable by the one or more processors, such as, but not limited to, an executable web browser (e.g., Google Chrome™, Apple Safari™, etc.), and additionally or alternatively, an executable application (e.g., application 212B) associated with the wireless communication system 100. In some cases, although not shown in FIG. 2 , the memory 212 may include one or more structured or unstructured data repositories or databases, and each of the UEs 104 may maintain one or more elements of device data in the one or more structured or unstructured data repositories or databases. For example, the element of device data may uniquely identify the UE 104 within the wireless communication system 100 and may include, but is not limited to, an Internet Protocol (IP) address assigned to the UE 104 or a media access control (MAC) layer address assigned to the UE 104.

[0027] Each of the UEs 104 may include an antenna unit, such as antenna unit 216A, configured to detect and / or receive data transmissions or beams / resources from at least the BS 103. In some examples, the antenna unit may include one or more antennas, each of which may detect one or more beams transmitted from the BS 103. As described herein, each of the UEs 104 may determine one or more measurements associated with each beam or resource transmitted from the BS 103, such as the RSTD, the RSRPP, the RSRP, a time difference between the received and transmitted measurements, and a location estimate of the corresponding UE 104. Additionally, each of the UEs 104 may include a display unit, such as display unit 216B, configured to present interface elements to a corresponding user, such as a user of the UE 104, and an input unit, such as input unit 216C, configured to receive input from the user (e.g., in response to interface elements presented via the display unit). By way of example, the display unit may include, but is not limited to, an LCD display unit or other suitable type of display unit, and the input unit 216C may include, but is not limited to, a keypad, a keyboard, a touch screen, voice-activated control technology, or any other suitable type of input unit. Furthermore, in additional aspects (not shown in FIG. 1), the functionality of the display unit and the input unit may be integrated into a single device, for example, in a pressure-sensitive touchscreen display unit that presents interface elements and receives input from a user.Additionally, each of the UEs 104 may also include a communication interface, e.g., a wireless transceiver device, such as communication interface 216D, coupled to a processor of the corresponding UE 104, such as processor 214, and configured by the processor to establish and maintain communication with a communication network via one or more communication protocols, such as WiFi, Bluetooth, NFC, a cellular communication protocol (e.g., LTE, CDMA, GSM, etc.), or any other suitable communication protocol.

[0028] Examples of UE 104 (e.g., UE 104A, UE 104B, UE 104C) may include, but are not limited to, personal computers, laptop computers, tablet computers, notebook computers, handheld computers, personal digital assistants, portable navigation devices, mobile phones, smartphones, wearable computing devices (e.g., smart watches, wearable activity monitors, wearable smart jewelry, and eyeglasses and other optical devices including optical head-mounted displays (OHMDs)), embedded computing devices (e.g., in communication with smart textiles or electronic fabrics), and any other type of computing device that may be configured to store data and software instructions, execute software instructions to perform operations, and / or display information on an interface device or unit, such as display unit 216B. In some examples, UE 104 may be a vehicle. In other examples, UE 104 may be an automated vehicle (e.g., a vehicle with autonomous driving capabilities). In some cases, the UE 104 may also establish communication with one or more additional computing systems or devices operating within the wireless communication system 100 via wired or wireless communication channels (e.g., via a communication interface using any suitable communication protocol).

[0029] In some examples, each of the UEs 104 may perform operations to determine measurements of one or more detected beams transmitted by the BS 103. Additionally, each of the UEs 104 may generate measurement data based on the determined measurements. For example, each element of the measurement data may include measurement information for a measurement determined at a particular time point when the measurement was made. Furthermore, each element may be associated with a timestamp generated by the corresponding UE 104. The timestamp may indicate the particular time point when the measurement was made. In some examples, each of the UEs 104 may implement a UE-assisted or UE-based positioning method to determine such measurements or generate such measurement data, such as a multi-cell round trip time (multi-RTT) positioning method, a downlink time difference of arrival (DL-TDOA) positioning method, and a downlink angle of departure (DL-AoD) positioning method. For example, while the BS 103 is operating in a wireless communication system such as New Radio (NR), one or more UEs 104, such as UE 104A, UE 104B, and UE 104C, may implement a UE-assisted or UE-based positioning method, such as a multi-cell round-trip time (multi-RTT) positioning method, a downlink time difference of arrival (DL-TDOA) positioning method, and a downlink angle of emission (DL-AoD) positioning method, to generate measurement data. In various cases, the measurement data may be utilized by the LMF computing system 102 to communicate additional data, such as assistance data, that each of the one or more UEs 104 may utilize to determine its own position.

[0030] In some cases, each of the UEs 104 may operate in a normal mode or a first mode associated with normal operation. While each of the UEs 104 is operating in the first mode, the corresponding UE 104, such as UE 104A, may perform operations to determine measurements of one or more detected beams transmitted by the BS 103 by utilizing a process associated with the first mode. In some examples, the process may be a newly deployed trained machine learning process.

[0031] In other cases, each of the UEs 104 may be configured to deploy or implement multiple processes for the purpose of determining the performance of one of the multiple processes. In such cases, each of the UEs 104 may implement or operate in a mode associated with determining the performance of one of the multiple processes by comparing data, such as measurement data generated from one process and measurement data generated from another process, such as a legacy process. For example, a particular UE 104, such as UE 104A, may be configured to deploy a first trained machine learning process and a second trained machine learning process. In such cases, both the first trained machine learning process and the second trained machine learning process are associated with determining measurements and / or generating measurement data for one or more beams transmitted from the BS 103. Additionally, while a particular UE 104 is operating in the second mode, the UE 104 may implement the first trained machine learning process to generate the first measurement data and the second trained machine learning process to generate the second measurement data. A particular UE 104 may determine the performance of the first trained machine learning model based on the first measurement data and the second measurement data. In some cases, a process utilized by one or more UEs 104 to determine the performance of another process may be designated as a legacy process by an operator of the wireless communication system 100.

[0032] In various cases, one or more UEs 104 may determine the performance of a particular process based on a quality or standard threshold. For example, a particular UE 104, such as UE 104A, may be configured to deploy a first trained machine learning process to generate first measurement data of a beam transmitted from BS 103 and a second trained legacy machine learning process to generate second measurement data of the beam. In addition, the first trained machine learning process may be identified by the LMF computing system 102 as a monitored process, and the quality or standard threshold may be a predetermined difference threshold that, if exceeded, may indicate insufficient performance of the first trained machine learning process. Furthermore, the particular UE 104 may determine a difference between the value of a minimum one element or measurement of the first measurement data and the value of a minimum one element or measurement of the second measurement data and compare the determined difference with the predetermined difference threshold. In an example where the determined difference exceeds the predetermined difference threshold, the particular UE 104 may determine that the first trained machine learning process is performing below the quality / standard threshold. In instances where the determined difference meets or falls below a predetermined difference threshold, the particular UE 104 may determine that the first trained machine learning process is operating at or above the quality / standard threshold, respectively.

[0033] In an example where a particular UE 104 determines that a process identified as a monitored process, such as a first trained machine learning process, is performing below a quality / standard threshold, the UE 104 may transmit report data to the LMF computing system 102 indicating that the monitored process is performing below a quality / standard threshold. In such an example, the LMF computing device 102 may modify or change the configuration of the particular UE 104. For example, the LMF computing system 102 may cause the particular UE 104 to operate in a third mode based on the report data. In some cases, while the particular UE 104 is operating in the third mode, the particular UE 104 may utilize a process utilized to determine the performance of the monitored process, such as a second trained legacy machine learning process, or another process designated by the operator of the wireless communications system 100 as robust and reliable, such as a third trained legacy machine learning process. For example, in response to the LMF computing device 102 determining, based on the reporting data, that a monitored process is performing below a quality / standard threshold, the LMF computing device 102 may operate a particular UE 104 in a third mode and automatically utilize, in place of the monitored process, a process utilized to determine the performance of the monitored process, such as a second trained legacy machine learning process, or another process designated by the operator of the wireless communications system 100 as robust and reliable, such as a third trained legacy machine learning process.

[0034] In an example where a particular UE 104 determines that a process identified as a monitored process, such as a first trained machine learning process, is performing at or above the quality / standard threshold, the UE 104 may transmit report data to the LMF computing device 102 indicating that the monitored process is performing at or above the quality / standard threshold. In such an example, the LMF computing system 102 may cause the particular UE 104 to perform the monitored process and generate measurement data. Accordingly, the LMF computing system 102 may provide location services to the particular UE 104 based on the measurement data associated with the monitored process. In some examples, the particular UE 104 may determine that a process identified as a monitored process, such as a first trained machine learning process, is performing at or above the quality / standard threshold. In such an example, the particular UE 104 may not generate and / or transmit to the LMF computing system report data indicating that the monitored process is performing at or above the quality / standard threshold. Instead, a particular UE 104 may initiate the implementation of an automatically monitored process only to generate measurement data that the LMF computing system 102 can utilize to provide location services to the particular UE 104.

[0035] In some cases, when the UE 104 determines that a monitored process, such as a first trained machine learning process, is performing below a quality / standard threshold, the UE 104 may automatically, and without receiving instructions from the LMF computing device 102, fall back to a default mode, a safe mode, or a third mode. For example, a particular UE 104, such as UE 104A, may determine that a first trained machine learning process is performing below a quality / standard threshold based on measurement data of the first trained machine learning process and measurement data of a second legacy process, such as a second trained legacy machine learning process. When the UE 104 determines that the first trained machine learning process is performing below a quality / standard threshold, the UE 104 may automatically fall back to and operate in the default mode, the safe mode, or the third mode. In such cases, the UE 104 may automatically fall back and operate in the default mode, the safe mode, or the third mode without communicating with the LMF computing system 102. Additionally, while the UE 104 is operating in the default mode, safe mode, or third mode, the particular UE 104 may deploy a process utilized to determine the performance of the monitored process, such as a second trained legacy machine learning process, or another process designated by the operator of the wireless communications system 100 as robust and reliable, such as a third trained legacy machine learning process, to determine measurements from one or more detected beams transmitted from the BS 103.

[0036] In various cases, each of the UEs 104 may determine measurements and / or generate measurement data associated with one or more beams detected by the corresponding UE 104 and transmitted by the BS 103 according to one or more monitoring parameters included in a reporting request received from the LMF computing system 102. For example, the LMF computing system 102 may send a reporting request to one or more UEs 104, such as UE 104A. As described herein, the reporting request may include parameter data, which may include one or more monitoring parameters, including timing parameters, resource parameters, and model parameters. In some examples, the reporting request may include data characterizing a quality / standard threshold, such as a difference threshold value. Additionally, the one or more UEs 104 may determine, in response to the one or more monitoring parameters, which processes to monitor (based on the model parameters), which resources, attributes, or data to measure from one or more detected beams transmitted from the BS 103 (based on the resource parameters), and the measurement time interval or period (based on the timing parameters). Moreover, based on such determination, one or more UEs 104 may implement an identified process, such as a first trained machine learning process, to make such measurements and generate corresponding measurement data. Furthermore, one or more UEs 104 may implement another or second process that generates measurement data that can be utilized to determine the performance of the identified monitored process. For example, one or more UEs 104 may implement a second process, such as a second trained legacy machine learning process, to make measurements and generate corresponding measurement data associated with one or more detected beams transmitted from the BS 103, depending on the timing and resource parameters. In some cases, the process or model parameters may also identify another or second process.

[0037] B. Computer-Implemented Techniques for Updating Machine Learning Processes As described herein, LMF computing systems 102, such as LMF computing system 102A, LMF computing system 102B, LMF computing system 102C, LMF computing system 102D, LMF computing system 102E, and LMF computing system 102F, may each be configured to monitor a new process, such as a first trained machine learning process, performed by one or more UEs 104, such as UE 104A, UE 104B, UE 104C, UE 104D, UE 104E, and UE 104F. Additionally, LMF computing system 102 may determine the performance of the new process based on measurement data of the new process as well as measurement data of a second process. As described herein, the second process may be a trained legacy machine learning process.

[0038] 3, the executed UE engine 208A may operate to generate a report request 302 that includes parameter data 304. As shown in FIG. 3, the parameter data 304 may include one or more monitoring parameters, such as timing parameters 304A, resource parameters 304B, and modeling or processing parameters 304C. As described herein, one or more parameters, such as timing parameters 304A (e.g., parameters indicating a time period or measurement period during which the corresponding UE 104 takes measurements from the new process and the legacy process), resource parameters 304B (e.g., parameters identifying one or more resources, attributes, or data that the corresponding UE 104 may measure from one or more detected beams transmitted from the BS 103), and modeling or processing parameters 304C (e.g., parameters that may identify at least a process to monitor, such as a new process, and a second process, such as a legacy process, from which measurement data is generated for comparison with the measurement data of the monitored process), may enable each of one or more UEs 104, such as UE 104A, to generate measurement data from the new process (e.g., a first new trained machine learning process) and the second process (e.g., a legacy process). Additionally, the measurement data associated with the new process and the measurement data associated with the second process may be compared by the UE 104 and / or the LMF computing system 102 to determine the performance of the new process.

[0039] For example, the executed UE engine 208A may access the data repository 204 to retrieve the UE data 206A for the UE 104A. As described herein, the UE data for the UE 104, such as the UE data 206A for the UE 104A, may include data associated with a new process that the executed UE engine 208A monitors, such as a new trained machine learning process, and data associated with another process, such as a legacy process, that the executed UE engine 208A utilizes to determine the performance of the new process. The data associated with the other process, such as the new process and the legacy process, may include data identifying the new process and the other process, and data identifying and characterizing parameters of each of the new process and the other process, such as model parameters in an example where one or both of the processes are trained machine learning processes. Additionally, the UE data for the UE 104, such as the UE data 206A for the UE 104A, may include data identifying the corresponding UE 104, such as the UE 104A (e.g., a corresponding serial number or identification number), and data indicating the performance status of one or more processes. Examples of performance statuses of one or more processes include a performance status indicating that the performance of a particular process is below a predetermined standard and a performance status indicating that the performance of a particular trained machine learning process is above or at a predetermined standard. Additionally, the executed UE engine 208A may generate parameter data 304 that includes one or more portions of the UE data 206A of the UE 104A. For example, the parameter data 304 may include data that identifies the UE 104A. Furthermore, the executed UE engine 208A may generate parameter data 304 that includes data based on one or more portions of the UE data of the UE 104, for example, one or more parameters that the UE engine 208A derives from the UE data 206A.

[0040] In some cases, the executed UE engine 208A may determine one or more parameters for a particular UE 104, such as the UE 104A, based on UE data of a new process monitored by the UE engine 208A (e.g., the UE data 206A of the UE 104A) and UE data of another process, such as a legacy process (e.g., the UE data 206A of the UE 104A). In such cases, a second process may be utilized by the UE engine 208A to determine the performance of the new process. For example, the UE engine 208A may determine timing parameters 304A based on data of model parameters of the new process and the legacy process. In some cases, the UE engine 208A may determine timing parameters 304A that cause a UE 104, such as the UE 104A, to configure the new process and the legacy process to generate comparable measurement data based on model parameters associated with measurement periods of the new process and the legacy process. For example, one or more elements of the measurement data for the new process and one or more elements of the measurement data for the legacy process may have matching timestamps or timestamps that are within a predetermined time threshold or margin.

[0041] Additionally, the UE engine 208A may generate a reporting request 302 and may package one or more portions of the parameter data 304 within the portions of the reporting request 302. Further, the executed UE engine 208A may transmit the reporting request 302 to the UE 104A. Although FIG. 3 illustrates the wireless communications system 100 including one LMF computing system 102, such as LMF computing system 102A, in communication with one UE 104, such as UE 104A, the LMF computing system 102 may transmit reporting requests, such as the reporting request 302, to any number of UEs 104. Further, the reporting request may be specific to a process that the LMF computing system 102 is monitoring on the corresponding UE 104. The LMF computing system 102 may begin monitoring a particular process on the corresponding UE 104 by sending a reporting request to the corresponding UE 104.

[0042] As described herein, an LMF computing system 102, such as LMF computing system 102A, may monitor the performance of a new process identified in a report request 302 based on first measurement data of the new process and second measurement data of the legacy process. The first measurement data and second measurement data may be transmitted from a corresponding UE 104 to the LMF computing system 102. Additionally, a corresponding UE 104, such as UE 104A, may generate first measurement data by applying the new process to one or more detected beams transmitted from the BS 103 and generate second measurement data by applying the legacy process to the detected one or more beams. Moreover, a UE 104, such as UE 104A, may apply the new process and the legacy process to the detected one or more beams in response to the parameter data 304 of the report request. Furthermore, a UE 104, such as UE 104A, may generate report data associated with the first measurement data and the second measurement data. In various examples, the report data may indicate performance of a monitored process, such as a first process, based on the first measurement data and the second measurement data. Figure 4 shows an example UE 104A generating report data 415. Although Figure 4 may show only UE 104A, any number of UEs 104 may perform the operations described herein to generate report data, such as report data 415. Each UE 104 that receives a report request from an LMF computing system 102, such as LMF computing system 102A, may each generate report data, such as report data 415, associated with the report request.

[0043] 4, a programmatic interface established and maintained by the UE 104A, such as an application programming interface (API) 402 of the UE 104A, may receive a report request 302 including parameter data 304. The parameter data 304 may include one or more parameters, such as timing parameters 304A, resource parameters 304B, and model parameters 304C. As described herein, the UE 104A may receive the report request 302 from an LMF computing system 102A, such as an executed UE engine 208A, over a communications network via a communication channel programmatically established between the API 402 and the executed UE engine 208A.

[0044] In various examples, one or more application programs 212B executed by the processor 214 of the UE 104A, such as the process module 404, analysis module 406, and notification module 408 of the UE 104A, may perform any of the example processes described herein to generate reporting data 415 indicative of the performance of a new process, such as a first new trained machine learning process. The executed UE engine 208A of the LMF computing system 102 may utilize the reporting data 415 obtained from the UE 104A to modify or change the configuration of the UE 104A. As an example, when executed by the processor 214 of the UE 104A, the executed process module 404 may perform an operation that causes parameter data 304 to be stored in the memory 212. In such an example, the parameter data 304 may include one or more parameters, such as timing parameters 304A, resource parameters 304B, and model parameters 304C.

[0045] Additionally, the executed process module 404 may perform an operation to access the memory 212 to obtain process data 411. Portions of the process data 411 may be associated with one or more processes, such as a new process and a legacy process, that the UE 104A (or any UE 104) may perform. Each of the one or more processes that the UE 104A may perform may be associated with generating measurement data from one or more beams detected by the antenna unit 216A and transmitted from the BS 103. Furthermore, the executed process module 404 may perform an operation to access the memory 212 to obtain parameter data 304. Based on the model parameters 304C of the report request, the process module 404 may identify the process that the LMF computing device 102A is monitoring, such as a new process, as well as additional processes for determining performance of the process that the LMF computing device 102A is monitoring. In some examples, a corresponding UE 104, such as the UE 104A, may utilize measurement data of the additional process, such as a legacy process, in determining performance of the monitored process. In other examples, the model parameters 304C may identify at least the process to be monitored, such as a new process, and may identify additional processes.

[0046] Further, the executed process module 404 may obtain a portion of process data 411 associated with the monitored process, such as a new process, and a portion of process data 411 associated with the additional process, such as a legacy process, based on the identified monitored process and the identified additional process. In some cases, the monitored process, such as a new process, is a new trained machine learning process. In such cases, the portion of process data 411 associated with the monitored process, such as a new process, may include one or more model parameters. Additionally, the process module 404 may deploy the new process according to the one or more model parameters included in the portion of associated process data 411. In other cases, in examples where the legacy process is a trained legacy machine learning process that the process module 404 may utilize to deploy the additional or legacy process, the portion of process data 411 associated with the legacy process includes one or more parameters, such as model parameters.

[0047] Based on the obtained portion of process data 411 associated with the new or monitored process, the obtained portion of process data 411 associated with the additional or legacy process, and the obtained parameter data 304, the executed process module 404 may apply the new process and the legacy process to beam data 412 of one or more beams detected by the antenna unit 216A and transmitted from the BS 103. In some examples, the executed process module 404 may apply the new process to the beam data 412 according to the portion of process data 411 associated with the new process and the parameter data 304. In such examples, the executed process module 404 may generate first measurement data 414A associated with the new process. The first measurement data 414A may include one or more measurements of the detected beams of the beam data 412. Additionally, the executed process module 404 may apply the legacy process to the beam data 412 according to the portion of process data 411 associated with the legacy process and the parameter data 304. Additionally, the executed process module 404 may generate second measurement data 414B associated with the legacy process. The second measurement data 414B may include one or more measurements of the detected beam of the beam data 412. In some cases, the process module 404 may store the first measurement data 414A and the second measurement data 414B in the memory 212.

[0048] As described herein, the parameter data 304 may enable measurement data generated by the monitored process and the additional process, such as first measurement data 414A and second measurement data 414B, to be compared for purposes of determining performance of the monitored process. For example, the parameter data 304 may include timing parameters 304A that identify a time period or measurement period during which the corresponding UE 104 takes measurements from the new process and the legacy process. In such a case, continuing the above example, the executed process module 404 may apply the new process to the beam data 412 in accordance with the portion of the process data 411 associated with the new process and for the identified period as indicated in the timing parameters 304A. Additionally, the executed process module 404 may apply the legacy process to the beam data 412 in accordance with the portion of the process data 411 associated with the legacy process and for the identified period as indicated in the timing parameters 304A. In this manner, the generated first and second measurement data of the new and legacy processes may each include one or more elements that may have a timestamp or a timestamp that is within a predetermined time threshold or margin. As described herein, each element of the one or more elements may be associated with a measurement of a beam in the beam data 412 determined by a corresponding process, such as a new process or a legacy process.

[0049] In another example, the parameter data 304 may include resource parameters 304B that identify one or more resources, attributes, or data that a corresponding process, such as a new process or a legacy process, may measure from the beam data 412 of one or more beams detected by the antenna unit 216A and transmitted from the BS 103. In such a case, such resources, attributes, or data may be associated with the assistance data, PFL ID(s), PRS resource ID, TRP, and PRS resource set ID. Additionally, following the above example, the executed process module 404 may apply the new process to the beam data 412 according to the portion of the process data 411 associated with the new process and for the identified resources as indicated in the resource parameters 304B. Moreover, the executed process module 404 may apply the legacy process to the beam data 412 according to the portion of the process data 411 associated with the legacy process and for the identified resources as indicated in the resource parameters 304B. In this manner, the generated first measurement data 414A and second measurement data 414B of the new process and the legacy process may each include one or more elements, and each of the one or more elements may be associated with the same identified resource. As described herein, the one or more elements may be further associated with measurements or measurement information of a beam of the beam data 412 determined by a corresponding process, such as the new process or the legacy process, and of the same identified resource. For example, based on the identified resource indicated in the resource parameters 304B, each of the one or more elements of the first measurement data 414A and the second measurement data 414B may include measurement information or characterize a measurement associated with an RSRPP.

[0050] In some examples, the resource parameter 304B may be associated with a particular positioning method (e.g., UE-assisted or UE-based). In such examples, the first measurement data 414A and the second measurement data 414B may be associated with the positioning method with which the resource parameter 304B is associated. For example, the report request 302 may include the resource parameter 304B associated with a DL-TDOA positioning method. In addition, the corresponding UE 104 may perform a new process to generate the first measurement data 414A and a legacy process to generate the second measurement data 414B. In such examples, the first measurement data 414A and the second measurement data 414B may each be associated with a DL-TDOA positioning method. For example, the first measurement data 414A and the second measurement data 414B may include data identifying RSTD, RSRP, and / or RSRPP.

[0051] 4 , the executed process module 404 may provide first measurement data 414A of the beam data 412 and second measurement data 414B of the beam data 412 as input to the executed analysis module 406. The executed analysis module 406 may perform an operation to compare the first measurement data 414A and the second measurement data 414B. For example, the executed analysis module 406 may access the memory 212 to obtain the first measurement data 414A and the second measurement data 414B. Additionally, the executed analysis module 406 may parse the first measurement data 414A to obtain one or more elements of the first measurement data 414A and parse the second measurement data 414B to obtain one or more elements of the second measurement data 414B. As described herein, each of one or more elements of the first measurement data 414A and the second measurement data 414B may be associated with a particular measurement made from a particular resource by a corresponding process, such as a new process or a legacy process. Additionally, the measurements may have been made by a corresponding process (e.g., a new process or a legacy process) on beam data 412 of one or more beams detected by antenna unit 216A and transmitted from BS 103. Furthermore, each of one or more elements of the first measurement data 414A and the second measurement data 414B may include a value of the associated measurement, and each of the first measurement data 414A and the second measurement data 414B may be associated with a timestamp indicating when the associated measurement was made by UE 104A using the new process or the legacy process.

[0052] Additionally, the executed analysis module 406 may compare the value of each of one or more elements of the first measurement data 414A with the value of each of one or more elements of the second measurement data 414B. The element values ​​of the first measurement data 414A compared by the executed analysis module 406 to the element values ​​of the second measurement data 414B may each be associated with a timestamp that matches or is within a predetermined time threshold or margin. Furthermore, the executed analysis module 406 may determine the difference between the values ​​and determine whether the determined difference exceeds the difference threshold. Based on the executed analysis module 406's determination of whether the determined difference exceeds the difference threshold, the executed analysis module 406 may determine whether the first process or new process associated with the first measurement data 414A is performing above or below a quality / standard threshold.

[0053] In some examples, the executed analysis module 406 may determine that the determined difference exceeds a difference threshold. In such examples, the executed analysis module 406 may determine that a first process, such as a new process, is performing below a quality / standard threshold. Additionally, the executed analysis module 406 may generate report data 415 indicating that a monitored process or a first process, such as a new process, is performing below a quality / standard threshold. Moreover, the executed analysis module 406 may store the report data 415 in the memory 212. In some cases, the executed notification module 408, when executed by the processor 214 of the UE 104A, may generate a notification message 416. Furthermore, the executed notification module may include one or more portions of the report data 415 indicating that a monitored process or a first process, such as a new process, is performing below a quality / standard threshold within a portion of the notification message 416. In such cases, the executed notification module 408 may transmit the notification message 416 to the LMF computing system 102A.

[0054] In other examples, the executed analysis module 406 may determine that the determined difference is less than or equal to the difference threshold. In such examples, the executed analysis module 406 may determine that a first process, such as a new process, is performing at or above the quality / standard threshold. Additionally, the executed analysis module 406 may generate report data 415 indicating that a monitored process or a first process, such as a new process, is performing at or above the quality / standard threshold. Moreover, the executed analysis module 406 may store the report data 415 in the memory 212. In some cases, the executed notification module 408, when executed by the processor 214 of the UE 104A, may generate a notification message 416. Furthermore, the executed notification module may include one or more portions of the report data 415 indicating that a monitored process or a first process, such as a new process, is performing at or above the quality / standard threshold within a portion of the notification message 416. In such cases, the executed notification module 408 may transmit the notification message 416 to the LMF computing system 102A.

[0055] In various examples, if the executed analysis module 406 determines that the determined difference exceeds the difference threshold, the executed analysis module 406 may generate report data 415. In such examples, if the executed analysis module 406 determines that the determined difference does not exceed the difference threshold, the executed analysis module 406 may generate report data 415. Alternatively, the executed notification module 408 may generate a notification message 416 for report data 415 indicating that a monitored process or a first process, such as a new process, is performing below a quality / standard threshold. Additionally, the executed notification module 408 may generate a notification message 416 for report data indicating that a monitored process or a first process, such as a new process, is performing above a quality / standard threshold.

[0056] An LMF computing system 102, such as LMF computing system 102A, may receive a notification message from one or more UAs 104 including report data indicating that a monitored process is performing below a quality / standard threshold (e.g., a notification message 416 including report data 415 indicating that a monitored process or a first process, such as a new process, is performing below a quality / standard threshold). The LMF computing system 102 may modify or change the configuration of the corresponding UE 104 based on the report data. As shown in FIG. 5, a UE 104A, such as an executed notification module 408, may send the notification message 416 across a communication network via a communication channel established between the executed notification module 408 and the API 502. As described herein, the notification message 416 may include report data 415 indicating that a monitored process or a first process, such as a new process, is performing below a quality / standard threshold. The API 502 of the server 202 can receive the notification message 416 and can route the notification message 416 to the executed UE engine 208A. The executed UE engine 208A can perform operations to parse the notification message 416 and obtain one or more portions of the reporting data 415. Further, the executed UE engine 208A can store the one or more portions of the reporting data 415 in corresponding portions of the data repository 204, such as the UE data store 206.

[0057] Additionally, the executed UE engine 208A may perform operations to modify or change the configuration of a corresponding UE 104, such as UE 104A, based on one or more portions of the reporting data. Referring to FIG. 5, the executed UE engine 208A may access the data repository 204 and retrieve one or more portions of the reporting data 415. The UE engine 208A may generate instructions 510 associated with the UE 104A (or any corresponding UE 104 of the retrieved one or more portions of the reporting data) based on the one or more portions of the reporting data 415. Additionally, the executed UE engine 208A may store the instructions 510 in a corresponding portion of the data repository 204, such as the UE data store 206. In some cases, the instructions 510 may cause the UE 104A to modify or change the configuration of the UE 104A (or any corresponding UE 104 of the reporting data). For example, one or more portions of the report data 415 may indicate that the performance of a first process, such as a new process, is below a quality / standard threshold (e.g., the difference between the first measurement data 414A and the second measurement data 414B of the new process exceeds a difference threshold). In such an example, the UE engine 208A may generate instructions 510 that may cause the UE 104A to switch to or operate in a default / safe mode. In some cases, while the UE 104A is operating in the default mode, the safe mode, or a third mode, the UE 104A may be configured to perform or deploy an additional process, such as a legacy process or another process that the operator of the wireless communications system 100 has designated as robust and reliable, such as a third trained legacy machine learning process. In such a case, the UE 104A may apply the additional or other process to generate measurement data that the LMF computing system 102 may utilize to provide location services for the UE 104A.

[0058] As described herein, UE 104A (or any other UE 104) may operate in a reporting mode while UE 104A (or any other UE 104) generates measurement data, such as measurement data 414, and report data, such as report data 415. Additionally, UE 104A (or any other UE 104) may be configured to operate in the reporting mode upon receiving a report request, such as report request 302, from an LMF computing system 102, such as LMF computing system 102A. Furthermore, when UE 104A receives or processes instruction 510, UE 104A may switch from the reporting mode to a default / safe mode.

[0059] Moreover, when executed by one or more processors of the server 202 of the LMF computing system 102A, the executed notification engine 520 may access a data repository to retrieve the instructions 510. Moreover, the executed notification engine 520 may generate a configuration message 522 and package one or more portions of the instructions 510 within portions of the configuration message 522. Furthermore, the executed notification engine 520 may send the configuration message 522 to the UE 104A over a communication network via a communication channel established between the UE 104A and the executed notification engine 520.

[0060] In some examples, an LMF computing system 102, such as LMF computing system 102A, may process measurement data of a monitored process and measurement data of another process, such as a legacy process, to determine the performance of the monitored process. In some cases, report data 415 may include measurement data of a monitored process, such as first measurement data 414A, which may be a new, trained machine learning process. In addition, report data 415 may include measurement data of a legacy process, such as second measurement data 414B. In addition, the measurement data of the new, trained machine learning process may include location estimates, while the measurement data of the legacy process may include raw data or measurements that an LMF computing system 102, such as an executed UE engine 208A, may utilize to generate a corresponding location estimate. In such cases, the LMF computing system 102 may compare the location estimates of the new, trained machine learning process with location estimates determined from the legacy process to determine the accuracy and performance of the new, trained machine learning process. Based on such determination, the LMF computing system 102 may generate instructions, such as instructions 510, to a corresponding UE 104, such as UE 104A. As described herein, the instructions may cause the corresponding UE 104 to operate in different modes, perform processes associated with the different modes, and generate measurement data. The measurement data may then be utilized by the LMF computing system 102 to provide location services to the corresponding UE.

[0061] For example, the LMF computing system 102 may determine a difference between a value associated with a location estimate of the new trained machine learning process and a value associated with a location estimate determined from the legacy process. In addition, the LMF computing system 102 may determine whether the determined difference exceeds a difference threshold. In instances where the determined difference exceeds the difference threshold, the LMF computing system 102 may generate instructions that cause the corresponding UE 104 to operate in a default / safe mode, as described herein.

[0062] In other examples, an LMF computing system 102, such as LMF computing system 102A, may update, for a particular UE 104, the status of a process running on the corresponding UE 104. In such examples, the process may be a process being monitored by the corresponding LMF computing system 102. Additionally, the process may be a new process, such as a new trained machine learning process. For example, with reference to FIG. 5, the LMF computing system 102A may receive a notification message 416 from the UE 104A that includes reporting data 415 indicating that the new process being monitored is performing at or above the quality / standard threshold. In such examples, the UE engine 208A may access the UE data 206A of the UE 104A and update data related to the status of the new process based on the reporting data 415 (e.g., when the new process is currently performing at or above the quality / standard threshold or when the UE 104A determines that the new process is performing at or above the quality / standard threshold). In another example, the LMF computing system 102A may receive a notification message 416 from the UE 104A that includes reporting data 415 indicating that a new process being monitored is performing below a quality / standard threshold. In such an example, the UE engine 208A may access the UE data 206A of the UE 104A and update data related to the status of the new process based on the reporting data 415 (e.g., when the new process is currently performing below the quality / standard threshold or when the UE 104A determines that the new process is performing below the quality / standard threshold).

[0063] 6, the API 602 of the UE 104A may receive a configuration message 522 from the UE 104A, such as the executed notification engine 520 of FIG. 5. The API 602 may route the configuration message 522 to the executed process module 404. In addition, the executed process module 404 may parse the configuration message 522 and obtain one or more portions of the instructions 510. Furthermore, the executed process module 404 may store the one or more portions of the instructions 510 in the memory 212.

[0064] In some examples, the executed process module 404 may configure the UE 104A according to the instructions 510. In such examples, the executed process module 404 may perform an operation to access the memory 212 to obtain the instructions 510. As described herein, the instructions 510 may include program instructions for the UE 104A to switch to or operate in a default / safe mode. In addition, the instructions 510 may identify a particular process associated with the default / safe mode. Further, while the UE 104A is operating in the default mode, the UE 104A may generate measurement data utilizing a process associated with the default / safe mode. In some cases, the particular process identified in the instructions 510 may be a process utilized to determine the performance of a legacy process, such as a legacy process or another process designated by an operator of the wireless communications system 100 as robust and reliable, such as a third trained legacy machine learning process.

[0065] In various cases, in response to the UE 104A determining that the performance of the monitored process or new process falls below a quality or standard threshold, the UE 104A may automatically fall back and operate in a default / safe mode. As described herein, the UE 104A (or any UE 104) may be operating in another mode, such as a second mode or a reporting mode, when the UE 104A determines whether the performance of the monitored process or new process falls below a quality or standard threshold. Thus, when the performance of the monitored process or new process is determined to fall below a quality or standard threshold, the UE 104A (or any UE 104) may automatically switch to the default / safe mode without receiving instructions 510 from the LMF computing system 102A (or any LMF computing system 102).

[0066] Further, while the UE 104A is operating based on the instructions 510 in the default / safe mode, the executed process module 404 may perform the process identified in the instructions 510 to generate the measurement data 604. For example, the executed process module 404 may identify a particular process, such as a legacy process or other process, based on the instructions 510. In addition, the executed process module 404 may access the memory 212 to obtain a portion of the process data 411 associated with the identified particular process. As described herein, the portion of the process data 411 associated with the identified particular process may include one or more parameters that enable the executed process module 404 to configure and deploy the identified particular process. For example, the particular process may be a trained legacy machine learning process, and the obtained portion of the process data 411 may include one or more model parameters of the trained legacy machine learning process. Thus, the executed process module 404 may configure and deploy the trained legacy machine learning process according to the one or more model parameters. Additionally, the executed process module 404 may apply the identified particular process to the beam data 312 of one or more beams detected by the antenna unit 216A and transmitted by the BS 103. Furthermore, the executed process module 404 may generate measurement data 604 based on the application of the identified particular process to the beam data 312. In some cases, the executed process module 404 may store the measurement data 604 in the memory 212.

[0067] As described herein, the LMF computing system 102A (or any LMF computing system 102) may utilize measurement data, such as measurement data 604, to provide location services for the UE 104A. For example, with reference to FIG. 6, the executed notification module 408 may access the memory 212 to obtain the measurement data 604. Additionally, the executed notification module 408 may generate a measurement message 606 and package one or more portions of the measurement data 604 within the portions of the measurement message 606. Further, the executed notification module 408 may transmit the measurement message 606 to the LMF computing system 102A. The executed LMF computing system 102A, such as the UE engine 208A, may parse the measurement message 606, obtain one or more portions of the measurement data 604 from the parsed measurement message 606, and perform operations to provide location services to the UE 104A based on the obtained one or more portions of the measurement data 604 (not shown in FIG. 6).

[0068] In some examples, the UE 104A may be configured to operate in one or more modes, such as a first mode or normal mode, a second mode or reporting mode, and a third mode or default / safe mode. Each mode may be associated with the deployment of one or more processes, such as a new process and a legacy process. In some cases, prior to receiving the report request 302, the UE 104A (or any UE 104) may be configured to operate in the first mode or normal mode. While the UE 104A is operating in the first mode or normal mode, the UE 104A may apply a first process, such as a new process, to one or more detected beams transmitted from the BS 103 according to one or more parameters of the first process included in the acquired corresponding portion of the process data 411. Additionally, the UE 104A may determine one or more measurements based on the application of the first process to one or more detected beams (e.g., beam data 412) and generate measurement data, such as first measurement data 414A, including the one or more measurements. Additionally, the UE 104A may communicate measurement data, such as the first measurement data 414A, to the LMF computing system 102A. The LMF computing system 102A may provide location services to the UE 104A based on the measurement data.

[0069] In other cases, in response to receiving the report request 302, the UE 104A (or any UE 104) may be configured to operate in a second mode or reporting mode. While the UE 104A is operating in the second mode or reporting mode, the UE 104A may apply a first process, such as a new process, and a second process, such as a legacy process, to one or more detected beams transmitted from the BS 103 according to one or more parameters of the first and second processes included in the obtained corresponding portions of the process data 411 and parameter data, such as the parameter data 304, included in the report request 302. As described herein, the UE 104A may determine one or more measurements based on the application of the first process to one or more detected beams (e.g., the beam data 412) and generate measurement data, such as first measurement data 414A, including the one or more measurements. Additionally, the UE 104A may determine one or more measurements based on application of the second process to one or more detected beams (e.g., beam data 412) and generate measurement data, such as second measurement data 414B, including the one or more measurements. Further, the UE 104A may communicate to the LMF computing system 102A an indication of the performance of the first process based on the measurement data of the first and second processes. The LMF computing system 102A may modify or change the configuration of the UE 104A based on the indication. As described herein, if the UE 104A is operating in another mode, such as a first mode or a normal mode, the UE 104A may switch to the reporting mode upon receiving the report request 302. In some cases, while the UE 104A is operating in the reporting mode, the UE 104A may simultaneously or in parallel deploy a first process, such as a new process (e.g., a new trained machine learning process), and a second process, such as a legacy process.

[0070] FIG. 7 is a flowchart of an example process 700 for determining performance of a process, such as a new process, deployed by a user equipment (UE) 104. For example, one or more LMF computing systems 102, such as LMF computing system 102A, may perform one or more of the steps of example process 700, as described below with reference to FIG. 7. With reference to FIG. 7, LMF computing system 102A may perform any of the processes described herein to send a report request 302 to UE 104A (e.g., in step 702 of FIG. 7). As described herein, report request 302 may include parameter data 304. In some cases, parameter data 304 may include one or more monitoring parameters, such as timing parameters 304A, resource parameters 304B, and modeling or processing parameters 304C. Additionally, executed UE engine 208A may utilize UE data 206A of UE 104A to determine one or more monitoring parameters included in parameter data 304. Additionally, as described herein, the UE data 206A of the UE 104A may include data associated with a new process that the executed UE engine 208A monitors, such as a new trained machine learning process, and data associated with another process, such as a legacy process, that the executed UE engine 208A utilizes to determine the performance of the new process. The data associated with the other process, such as the new process and the legacy process, may include data identifying the new process and the other process, and data identifying and characterizing parameters of each of the new process and the other process, such as model parameters in examples where one or both of the processes are trained machine learning processes. Additionally, the UE data 206A of the UE 104A may include data identifying the UE 104A (e.g., a corresponding serial number or identification number) and data indicative of the performance status of one or more processes.Examples of performance statuses of one or more processes include a performance status indicating that the performance of a particular process is below a predetermined standard and a performance status indicating that the performance of a particular trained machine learning process is below a predetermined standard.

[0071] For example, the executed UE engine 208A may access the data repository 204 and obtain the UE data 206A of the UE 104A. Additionally, the executed UE engine 208A may generate parameter data 304 including one or more portions of the UE data 206A of the UE 104A. For example, the parameter data 304 may include data identifying the UE 104A. Furthermore, the executed UE engine 208A may generate parameter data 304 including data based on one or more portions of the UE data 206A of the UE 104. For example, one or more parameters derived by the UE engine 208A from the UE data 206A, such as timing parameters 304A. Additionally, the executed UE engine 208A may generate a reporting request 302 and package one or more portions of the parameter data 304 within the portions of the reporting request 302. Furthermore, the executed UE engine 208A may send the reporting request 302 to the API 402 of the UE 104A.

[0072] In response to receiving the reporting request 302, the UE 104A may operate in a reporting mode, and the executed process module 404 may perform operations as described herein in accordance with the reporting request 302 to generate measurement data, such as first measurement data 414A and second measurement data 414B. Additionally, while the UE 104A is operating in the reporting mode, the executed process module 404 may determine the performance of a first process identified in the reporting request 302, such as a new trained machine learning process, based on the measurement data. The executed process module 404 may generate reporting data 415 based on the determined performance, which may be indicative of the performance of the first process. In some cases, the reporting data 415 may indicate a comparison between the measurement data generated by the first process, such as the first measurement data 414A, and the measurement data of a second process utilized to determine the performance of the first process. In such cases, the comparison may be indicative of the performance of the first process. Additionally, the process module 404 being executed may send reporting data 415 to the LMF computing system 102A, such as the API 502.

[0073] Referring again to FIG. 7 , the LMF computing system 102A may perform any of the processes described herein to receive reporting data 415 from the UE 104A (e.g., in step 704 of FIG. 7 ). Additionally, the LMF computing system 102A may generate instructions to cause the user equipment to perform at least one of a first process and a second process based on the reporting data 415 (e.g., in step 706 of FIG. 7 ). For example, the UE 104A, such as the executed notification module 408, may send a notification message 416 across the communication network via a communication channel established between the executed notification module 408 and the API 502. As described herein, the notification message 416 may include reporting data 415 indicating performance of the monitored process or the first process, such as a new process. The API 502 of the server 202 may receive the notification message 416 and route the notification message 416 to the executed UE engine 208A. The executed UE engine 208A may perform operations to parse the notification message 416 and obtain one or more portions of the reporting data 415. Additionally, the executed UE engine 208A may generate instructions 510 based on one or more portions of the reporting data 415.

[0074] In some cases, the reporting data 415 may indicate that the first process is performing below a quality / standard threshold. The executed UE engine 208A may generate instructions 510 associated with the UE 104A based on the reporting data 415. The instructions 510 may cause the UE 104A to modify or change a configuration of the UE 104A (or any corresponding UE 104 in the reporting data). For example, the instructions 510 may cause the UE 104A to switch from a reporting mode to a default / safe mode. In such cases, while the UE 104A is operating in the default mode, safe mode, or third mode, the UE 104A may be configured to perform or deploy an additional process, such as a legacy process or another process designated by the operator of the wireless communications system 100 as robust and reliable, such as a third trained legacy machine learning process.

[0075] In other cases, the reporting data 415 may indicate that the first process is performing above a quality / standard threshold. The executed UE engine 208A may generate instructions 510 associated with the UE 104A based on the reporting data 415. The instructions 510 may cause the UE 104A to modify or change a configuration of the UE 104A (or any corresponding UE 104 in the reporting data). For example, the instructions 510 may cause the UE 104A to switch from the reporting mode to the first mode or the normal mode. In such a case, while the UE 104A is operating in the normal mode or the first mode, the UE 104A may be configured to implement or deploy a first process, such as a new trained legacy machine learning process.

[0076] Additionally, the LMF computing system 102A may send the instructions 510 to the UE 104A (e.g., step 710 of FIG. 7 ). When executed by one or more processors of the server 202 of the LMF computing system 102A as described herein, the executed notification engine 520 may generate a configuration message 522 and may package one or more portions of the instructions 510 within portions of the configuration message 522. Further, the executed notification engine 520 may send the configuration message 522 to the UE 104A over a communication network via a communication channel established between the UE 104A and the executed notification engine 520. Additionally, the API 602 of the UE 104A may receive the configuration message 522 from the UE 104A, such as the executed notification engine 520 of FIG. 5 . The API 602 may route the configuration message 522 to the executed process module 404. Additionally, the executed process module 404 may parse the configuration message 522 to obtain one or more portions of the instructions 510 .

[0077] In the event that the instructions 510 are associated with reporting data 415 indicating that the first process is performing below a quality / standard threshold, the executed process module 404 may cause the UE 104A to switch from operating in the reporting mode to operating in a default / safe mode. As described herein, the default / safe mode may be associated with a particular process that the UE 104A may utilize for measurement data, such as a legacy process or another process that an operator of the wireless communications system 100 has designated as robust and reliable, such as a third trained legacy machine learning process. Additionally, while the UE 104A is operating in the default / safe mode, the UE 104A may perform operations described herein to generate measurement data, such as measurement data 604, that the LMF computing system 102A may utilize to provide location services for the UE 104A.

[0078] For example, the executed process module 404 may identify a particular process associated with the default / safe mode based on one or more portions of the instructions 510. Additionally, the executed process module 404 may access the memory 212 to retrieve a portion of the process data 411 associated with the identified particular process. As described herein, the portion of the process data 411 associated with the identified particular process may include one or more parameters that enable the executed process module 404 to configure and deploy the identified particular process. For example, the particular process may be a trained legacy machine learning process, and the retrieved portion of the process data 411 may include one or more model parameters of the trained legacy machine learning process. Thus, the executed process module 404 may configure and deploy the trained legacy machine learning process according to the one or more model parameters. Furthermore, the executed process module 404 may apply the identified particular process to beam data 312 of one or more beams detected by the antenna unit 216A and transmitted by the BS 103. Additionally, the executed process module 404 may generate metrology data 604 based on the application of the identified particular process to the beam data 312 .

[0079] As described herein, the LMF computing system 102A may obtain and utilize measurement data, such as measurement data 604, to provide location services for the UE 104A. For example, the executed notification module 408 may access the memory 212 to obtain the measurement data 604. In addition, the executed notification module 408 may generate a measurement message 606 and package one or more portions of the measurement data 604 within the portions of the measurement message 606. Further, the executed notification module 408 may transmit the measurement message 606 to the LMF computing system 102A. The executed LMF computing system 102A, such as the UE engine 208A, may parse the measurement message 606, obtain one or more portions of the measurement data 604 from the parsed measurement message 606, and perform operations to provide location services to the UE 104A based on the obtained one or more portions of the measurement data 604 (not shown in FIG. 6 ).

[0080] In the event that the instructions 510 are associated with reporting data 415 indicating that the first process is performing above a quality / standard threshold, the executed process module 404 may cause the UE 104A to switch from operating in the reporting mode to operating in the first mode or the normal mode. As described herein, the first mode / normal mode may be associated with a particular process, such as a first process, that the UE 104A may utilize for the measurement data. In some cases, the first process may be a new trained machine learning process. Additionally, while the UE 104A is operating in the first mode / normal mode, the UE 104A may perform operations described herein to generate measurement data, such as first measurement data 414A, that the LMF computing system 102A may utilize to provide location services for the UE 104A.

[0081] For example, the executed process module 404 may determine that the first process is associated with the first mode / normal mode based on one or more portions of the instructions 510. Additionally, the executed process module 404 may access the memory 212 to obtain a portion of the process data 411 associated with the first process. As described herein, the portion of the process data 411 associated with the first process may include one or more parameters that enable the executed process module 404 to configure and deploy the first process. For example, the first process may be a trained legacy machine learning process, and the obtained portion of the process data 411 may include one or more model parameters of a new trained legacy machine learning process. Thus, the executed process module 404 may configure and deploy a new trained machine learning process according to the one or more model parameters. Furthermore, the executed process module 404 may apply the identified first process to beam data 312 of one or more beams detected by the antenna unit 216A and transmitted by the BS 103. Additionally, the executed process module 404 may generate first measurement data 414A based on application of the first process to the beam data 312. In some cases, the executed process module 404 may store in the memory 212 the measurement data 414A generated while the UE 104A is operating in the first mode.

[0082] As described herein, the LMF computing system 102A may obtain and utilize measurement data, such as first measurement data 414A, to provide location services for the UE 104A. For example, the executed notification module 408 may access the memory 212 to obtain measurement data of a first process generated while the UE 104A was operating in a first mode, such as the measurement data 414A. In addition, the executed notification module 408 may generate a measurement message, such as a measurement message 606, and package one or more portions of the obtained measurement data within the measurement message. Further, the executed notification module 408 may transmit the measurement message to the LMF computing system 102A. The LMF computing system 102A, such as the execution UE engine 208A, may parse the measurement message, obtain one or more portions of the measurement data from the parsed measurement message 606, and perform operations to provide location services at the UE 104A based on the obtained one or more portions of the measurement data.

[0083] 8 is a flowchart of an example process 800 for generating reporting data according to an example embodiment. For example, one or more UEs 104, such as UE 104A, may perform one or more of the steps of the example process 800, as described below with reference to FIG. 8. With reference to FIG. 8, a first UE 104A of the plurality of UEs 104 (or any UE 104) may perform any of the processes described herein to obtain (e.g., in step 802 of FIG. 8) from the LMF computing system 102 a reporting request 302 including a first data set including one or more monitoring parameters. As described herein, the reporting request 302 may include parameter data 304. In some cases, the parameter data 304 may include one or more monitoring parameters, such as timing parameters 304A, resource parameters 304B, and modeling or processing parameters 304C.

[0084] Based on the report request 302, the UE 104A may generate one or more elements of the first measurement data 414A and utilize the first trained machine learning process identified in the report request 302 (e.g., step 804 of FIG. 8 ). As described herein, the first trained machine learning process may be a new process identified in the report request 302, such as from the modeling or processing parameters 304C of the report request 302. Additionally, the executed process module 404 may identify the first trained machine learning process from the report request 302 and access the memory 212 to obtain a portion of the process data 411 associated with the first trained machine learning process. In some cases, the portion of the process data 411 associated with the first trained machine learning process may include one or more modeling parameters. Based on the obtained portion of process data 411 associated with the first trained machine learning process and the parameter data 304 of the report request 302, the executed process module 404 may apply the first trained machine learning process to beam data 412 of one or more beams detected by the antenna unit 216A and transmitted from the BS 103. Further, the executed process module 404 may generate first measurement data 414A based on the application of the first trained machine learning process to the beam data 412.

[0085] Additionally, the UE 104A may generate one or more elements of the second measurement data 414B and utilize a second process while the UE 104A generates one or more elements of the first measurement data 414A based on the report request 302 (e.g., step 806 of FIG. 8 ). As described herein, the second process may be a legacy process and / or a trained machine learning process. Additionally, the second process may be utilized to determine the performance of the first trained machine learning process. In some cases, the report request 302 may identify the second process. In such a case, the executed process module 404 may identify the second process from the report request 302 and access the memory 212 to obtain a portion of the process data 411 associated with the second process. In some cases, the portion of the process data 411 associated with the second process may include one or more parameters that the executed process module 404 may utilize to develop the second process. The executed process module 404 may apply the second process to the beam data 412 based on the obtained portion of the process data 411 associated with the second process and the parameter data 304 of the report request 302. Further, the executed process module 404 may generate second measurement data 414B based on the application of the second process to the beam data 412.

[0086] Additionally, the UE 104A may compare the first measurement data 414A and the second measurement data 414B (e.g., step 808 of FIG. 8 ). As described herein, the UE 104A may determine the performance of the first trained machine learning process based on the comparison. For example, the executed analysis module 406 may perform an operation to compare the first measurement data 414A and the second measurement data 414B. For example, the executed analysis module 406 may parse the first measurement data 414A to obtain one or more elements of the first measurement data 414A and parse the second measurement data 414B to obtain one or more elements of the second measurement data 414B. Each of one or more elements of the first measurement data 414A and the second measurement data 414B may include a value of an associated measurement, and each of the first measurement data 414A and the second measurement data 414B may be associated with a timestamp indicating the time the associated measurement was made by the UE 104A using the first trained machine learning process and the second process, respectively. Additionally, the executed analysis module 406 may compare the values ​​of the elements of the first measurement data 414A with the values ​​of the elements of the second measurement data 414B, each associated with a timestamp that matches or falls within a predetermined time threshold or margin. Moreover, the executed analysis module 406 may determine a difference between the values ​​and determine whether the determined difference exceeds a difference threshold. Furthermore, based on the executed analysis module 406 determining whether the determined difference exceeds a difference threshold, the executed analysis module 406 may determine whether the first trained machine learning process associated with the first measurement data 414A is performing above or below a quality / standard threshold. For example, the executed analysis module 406 determines that the determined difference is less than or equal to a difference threshold. In such a case, the executed analysis module 406 may determine that the first trained machine learning process is performing at or above a quality / standard threshold. In another example, the executed analysis module 406 determines that the determined difference is greater than a difference threshold.In such cases, the executed analysis module 406 may determine that the first trained machine learning process is performing below a quality / standard threshold.

[0087] Implementation examples are further described in the following numbered clauses. Article 1. a non-transitory machine-readable storage medium storing instructions; at least one processor coupled to a non-transitory machine-readable storage medium, wherein the at least one processor executes instructions to: causing a user equipment to transmit a report request regarding positioning data, and receiving the report request by the user equipment; performing a first process for generating first positioning data and a second process for generating second positioning data; and generating report data indicative of a comparison between the first positioning data and the second positioning data; receiving report data from the user equipment; An apparatus configured to generate and send to the user equipment instructions that cause the user equipment to perform at least one of the first process or the second process based on the report data. Clause 2. The apparatus of clause 1, wherein the report request includes timing parameters specifying a time interval for the user equipment to operate in the first mode. Clause 3. The apparatus of any of clauses 1-2, wherein the reporting request includes resource parameters that identify one or more resources from which the user equipment is to generate positioning data. Clause 4. The apparatus of clause 3, wherein the one or more resources include resources selected from the group including resources associated with an Assistance Data ID, resources associated with a Positioning Frequency Layer (PFL) ID, resources associated with a Transmit Receive (TRP) ID, resources associated with a Positioning Reference Signal (PRS) Set ID, a set of Positioning Reference Signal (PRS) resources, or a combination thereof. Clause 5. The apparatus of any of clauses 1 to 4, wherein the first positioning data is associated with a first subset of resources and the second positioning data is associated with a second subset of resources. Clause 6. The apparatus of clause 5, wherein the second subset of resources includes the first subset of resources. Clause 7. The apparatus of any of clauses 1 to 6, wherein the first positioning data and the second positioning data each include one or more elements associated with the same resource. Article 8. The reporting request is sent to the user equipment: generating a first timestamp associated with a first element of the one or more elements of the first positioning data, the first element corresponding to the second element, and a second timestamp associated with a second element of the one or more elements of the second positioning data; Comparing the first timestamp to the second timestamp; determining that the first timestamp and the second timestamp are within a predetermined time interval; 8. The apparatus of any one of clauses 1 to 7, wherein the apparatus generates report data based on the determination. Clause 9. The reporting request further requires the user equipment to: 9. The apparatus of clause 8, for determining that the first timestamp and the second timestamp match. Clause 10. The reporting request further requires the user equipment to: An apparatus as described in any of clauses 1 to 9, wherein a first process is utilized to generate one or more elements of first positioning data and, simultaneously, a second process is utilized to generate one or more elements of second positioning data. Article 11. The reporting request further requires the user equipment to: comparing the first positioning data with the second positioning data; determining, based on the comparison, that a difference between a first element of the first positioning data and a corresponding first element of the second positioning data satisfies a difference threshold; 11. The apparatus of clause 10, causing the apparatus to transmit report data based on a determination that the difference satisfies a difference threshold. Clause 12. The user equipment operates in a second mode and performs a first process to generate first positioning data before receiving a report request from the device, and the receipt of the report request by the user equipment causes the user equipment to: Switching from operation in the second mode to operation in the first mode, determining that at least a first element of the first positioning data and a corresponding first element of the second positioning data are within a predetermined time margin; comparing at least a first element of the first positioning data with a corresponding first element of the second positioning data; 12. The apparatus of clause 11, wherein, in the condition where the comparison is greater than a predetermined threshold, the apparatus switches to operate in a third mode to generate additional positioning data. Clause 13. The apparatus of clause 12, wherein while the user equipment is operating in the third mode, the user equipment performs a second process to generate additional positioning data. Clause 14. The apparatus of clause 12, wherein while the user equipment is operating in the third mode, the user equipment performs a third process to generate additional positioning data. Clause 15. The apparatus of any of clauses 1 to 14, wherein the first process is a trained machine learning process. Clause 16. The apparatus of any of clauses 1 to 15, wherein the second process is a trained machine learning process. Clause 17. An apparatus described in any of clauses 1 to 16, wherein the report request includes model parameters, the model parameters identifying one or more processes to monitor, the one or more processes including a first process and a second process. Clause 18. A non-transitory machine-readable storage medium storing instructions that, when executed by at least one processor of a location server, cause the at least one processor to: transmitting a report request regarding positioning data to a user equipment, wherein receiving the report request by the user equipment includes: performing a first process for generating first positioning data and a second process for generating second positioning data; and causing the user equipment to perform operations in a first mode, including generating report data indicative of a comparison between the first positioning data and the second positioning data; receiving report data from a user equipment; and generating and transmitting to the user equipment instructions that cause the user equipment to perform at least one of a first process and / or a second process based on the report data. Clause 19. The non-transitory machine-readable storage medium of clause 18, wherein the report request includes timing parameters specifying a time interval for the user equipment to operate in the first mode. Clause 20. A non-transitory machine-readable storage medium according to any one of clauses 18 to 19, wherein the reporting request includes resource parameters that identify one or more resources from which the user equipment will generate positioning data. Clause 21. The non-transitory machine-readable storage medium of clause 20, wherein the one or more resources include resources selected from the group including resources associated with an Assistance Data ID, resources associated with a Positioning Frequency Layer (PFL) ID, resources associated with a Transmit Receive (TRP) ID, resources associated with a Positioning Reference Signal (PRS) Set ID, a set of Positioning Reference Signal (PRS) resources, or a combination thereof. Clause 22. A non-transitory machine-readable storage medium according to any of clauses 18 to 22, wherein the first positioning data is associated with a first subset of resources and the second positioning data is associated with a second subset of resources. Clause 23. The non-transitory machine-readable storage medium of clause 22, wherein the second subset of resources includes the first subset of resources. Clause 24. A non-transitory machine-readable storage medium according to any one of clauses 18 to 23, wherein the first positioning data and the second positioning data each include one or more elements associated with the same resource. Article 25. The reporting request is sent to the user equipment: generating a first timestamp associated with a first element of the one or more elements of the first positioning data, the first element corresponding to the second element, and a second timestamp associated with a second element of the one or more elements of the second positioning data; Comparing the first timestamp to the second timestamp; determining that the first timestamp and the second timestamp are within a predetermined time interval; 25. A non-transitory machine-readable storage medium according to any one of clauses 18 to 24, which generates report data based on the determination. Article 26. The reporting request further requires the user equipment to: 26. The non-transitory machine-readable storage medium of clause 25, for determining that a first timestamp and a second timestamp match. Article 27. The reporting request further requires the user equipment to: 27. A non-transitory machine-readable storage medium according to any one of clauses 18 to 26, wherein a first process is used to generate one or more elements of first positioning data and, simultaneously, a second process is used to generate one or more elements of second positioning data. Article 28. The reporting request further requires the user equipment to: comparing the first positioning data with the second positioning data; determining, based on the comparison, that a difference between a first element of the first positioning data and a corresponding first element of the second positioning data satisfies a difference threshold; 28. The non-transitory machine-readable storage medium of clause 27, causing the device to transmit report data based on a determination that the difference satisfies a difference threshold. Clause 29. The user equipment operates in a second mode and performs a first process to generate first positioning data before receiving a report request from the device, and the receipt of the report request by the user equipment causes the user equipment to: Switching from operation in the second mode to operation in the first mode, determining that at least a first element of the first positioning data and a corresponding first element of the second positioning data are within a predetermined time margin; comparing at least a first element of the first positioning data with a corresponding first element of the second positioning data; 29. The non-transitory machine-readable storage medium of clause 28, wherein, in a condition where the comparison is greater than a predetermined threshold, the non-transitory machine-readable storage medium switches to operate in a third mode to generate additional positioning data. Clause 30. The non-transitory machine-readable storage medium of clause 29, wherein while the user equipment is operating in the third mode, the user equipment performs a second process to generate additional positioning data. Clause 31. The non-transitory machine-readable storage medium of clause 29, wherein while the user equipment is operating in the third mode, the user equipment performs a third process to generate additional positioning data. Clause 32. The non-transitory machine-readable storage medium of any of clauses 18 to 31, wherein the first process is a trained machine learning process. Clause 33. The non-transitory machine-readable storage medium of any of clauses 18 to 32, wherein the second process is a trained machine learning process. Clause 34. A non-transitory machine-readable storage medium according to any of clauses 18 to 33, wherein the report request includes model parameters, the model parameters identifying one or more processes to monitor, the one or more processes including a first process and a second process. Clause 35. A computer-implemented method comprising: transmitting, by a processor of the location server, a report request regarding the positioning data to the user equipment, wherein receiving the report request by the user equipment includes: performing a first process for generating first positioning data and a second process for generating second positioning data; and causing the user equipment to perform operations in a first mode, including generating report data indicative of a comparison between the first positioning data and the second positioning data; receiving, by a processor, report data from the user equipment; and generating and transmitting to the user equipment instructions, based on the report data, that cause the user equipment to perform at least one of the first process and / or the second process. Clause 36. The computer-implemented method of clause 35, wherein the report request includes timing parameters specifying a time interval for the user equipment to operate in the first mode. Clause 37. The computer-implemented method of any of clauses 35-36, wherein the reporting request includes resource parameters that identify one or more resources from which the user equipment is to generate positioning data. Clause 38. The computer-implemented method of clause 37, wherein the one or more resources include resources selected from the group including resources associated with aiding, resources associated with a positioning frequency layer (PFL), resources associated with a transmit receive (TRP) ID, resources associated with a positioning reference signal (PRS) resource, or a combination thereof. Clause 39. The computer-implemented method of any of clauses 35-38, wherein the first positioning data is associated with a first subset of resources and the second positioning data is associated with a second subset of resources. Clause 40. The computer-implemented method of clause 39, wherein the second subset of resources includes the first subset of resources. Article 41. The reporting request is sent to the user equipment: generating a first timestamp associated with a first element of the one or more elements of the first positioning data, the first element corresponding to the second element, and a second timestamp associated with a second element of the one or more elements of the second positioning data; Comparing the first timestamp to the second timestamp; determining that the first timestamp and the second timestamp are within a predetermined time interval; 41. The computer-implemented method of any of clauses 35-40, generating report data based on the determination. Clause 42. The computer-implemented method of clause 41, wherein the report request further causes the user equipment to determine that the first timestamp and the second timestamp match. Clause 43. The computer-implemented method of clauses 35-42, wherein the report request further causes the user equipment to generate one or more elements of the first positioning data using a first process and simultaneously generate one or more elements of the second positioning data using a second process. Clause 44. The reporting request may further require the user equipment: comparing the first positioning data with the second positioning data; determining, based on the comparison, that a difference between a first element of the first positioning data and a corresponding first element of the second positioning data satisfies a difference threshold; 44. The computer-implemented method of claim 43, causing a processor of the location server to transmit the report data based on a determination that the difference satisfies a difference threshold. Clause 45. Before receiving a report request from the processor of the location server, the user equipment operates in a second mode, and the receipt of the report request by the user equipment causes the user equipment to: Switching from operation in the second mode to operation in the first mode, determining that at least a first element of the first positioning data and a corresponding first element of the second positioning data are within a predetermined time margin; comparing at least a first element of the first positioning data with a corresponding first element of the second positioning data; 45. The computer-implemented method of claim 44, wherein, in a condition where the comparison is greater than a predetermined threshold, switching to operate in a third mode to generate additional positioning data. Clause 46. The computer-implemented method of clause 45, wherein while the user equipment is operating in the third mode, the user equipment performs a second process to generate additional positioning data. Clause 47. The computer-implemented method of clause 45, wherein while the user equipment is operating in the third mode, the user equipment performs a third process to generate additional positioning data. Clause 48. The computer-implemented method of any of clauses 35 to 47, wherein the first process is a trained machine learning process. Clause 49. The computer-implemented method of any of clauses 35 to 48, wherein the second process is a trained machine learning process. Clause 50. The computer-implemented method of any of clauses 35-49, wherein the report request includes model parameters, the model parameters identifying one or more processes to monitor, the one or more processes including a first process and a second process. Article 51. means for transmitting, by a processor of the location server, a report request for positioning data to a user equipment, wherein receiving the report request by the user equipment comprises: performing a first process for generating first positioning data and a second process for generating second positioning data; and generating report data indicative of a comparison between the first positioning data and the second positioning data; means for receiving, by a processor, report data from the user equipment; and means for generating and transmitting to the user equipment instructions that cause the user equipment to perform at least one of the first process and / or the second process based on the reporting data. Clause 52. The positioning computing device of clause 51, wherein the report request includes timing parameters specifying a time interval for the user equipment to operate in the first mode. Clause 53. A positioning computing device according to any of clauses 51 to 52, wherein the reporting request includes resource parameters identifying one or more resources from which the user equipment will generate positioning data. Clause 54. The positioning computing device of clause 53, wherein the one or more resources include resources selected from the group including resources associated with assistance, resources associated with a positioning frequency layer (PFL), resources associated with a transmit receive (TRP) ID, resources associated with a positioning reference signal (PRS) resource, or a combination thereof. Clause 55. A positioning computing device according to any of clauses 51 to 54, wherein the first positioning data is associated with a first subset of resources and the second positioning data is associated with a second subset of resources. Clause 56. The positioning computing device method of clause 55, wherein the second subset of resources includes the first subset of resources. Article 57. The reporting request is sent to the user equipment: generating a first timestamp associated with a first element of the one or more elements of the first positioning data, the first element corresponding to the second element, and a second timestamp associated with a second element of the one or more elements of the second positioning data; Comparing the first timestamp to the second timestamp; determining that the first timestamp and the second timestamp are within a predetermined time interval; 57. A positioning computing device as described in any of clauses 51 to 56, causing reporting data to be generated based on the determination. Clause 58. The positioning computing device of clause 57, wherein the report request further causes the user equipment to determine that the first timestamp and the second timestamp match. Clause 59. A positioning computing device as described in any of clauses 51 to 58, wherein the reporting request further causes the user equipment to generate one or more elements of first positioning data using a first process and simultaneously generate one or more elements of second positioning data using a second process. Clause 60. The reporting request may further require the user equipment: comparing the first positioning data with the second positioning data; determining, based on the comparison, that a difference between a first element of the first positioning data and a corresponding first element of the second positioning data satisfies a difference threshold; 60. The positioning computing device of clause 59, causing the processor of the location server to transmit the report data based on a determination that the difference satisfies the difference threshold. Clause 61. Before receiving a report request from the processor of the location server, the user equipment operates in a second mode, and the receipt of the report request by the user equipment causes the user equipment to: Switching from operation in the second mode to operation in the first mode, determining that at least a first element of the first positioning data and a corresponding first element of the second positioning data are within a predetermined time margin; comparing at least a first element of the first positioning data with a corresponding first element of the second positioning data; 61. The positioning computing device of clause 60, wherein, in the condition where the comparison is greater than a predetermined threshold, the positioning computing device switches to operate in a third mode to generate additional positioning data. Clause 62. The positioning computing device of clause 61, wherein while the user equipment is operating in the third mode, the user equipment performs a second process to generate additional positioning data. Clause 63. The positioning computing device of clause 61, wherein while the user equipment is operating in the third mode, the user equipment performs a third process to generate additional positioning data. Clause 64. A positioning computing device according to any of clauses 51 to 63, wherein the first process is a trained machine learning process. Clause 65. A positioning computing device according to any of clauses 51 to 64, wherein the second process is a trained machine learning process. Clause 66. A positioning computing device as described in any of clauses 51 to 65, wherein the report request includes model parameters, the model parameters identifying one or more processes to monitor, the one or more processes including a first process and a second process. Article 67. a non-transitory machine-readable storage medium storing instructions; at least one processor coupled to a non-transitory machine-readable storage medium, wherein the at least one processor executes instructions to: Sending a request for reporting positioning data to the user equipment; receiving report data indicative of a comparison between the first positioning data generated from the first process and the second positioning data generated from the second process; The apparatus is configured to generate and send to the user equipment instructions that cause the user equipment to perform a first process based on the reporting data. Clause 68. At least one processor executes instructions to further: receiving third positioning data from the user equipment; The apparatus described in clause 67, configured to generate and send to the user equipment second instructions, based on the third positioning data, that cause the user equipment to perform a second process. Clause 69. The apparatus of any of clauses 67-68, wherein the report request includes timing parameters specifying a time interval for the user equipment to operate in the first mode. Clause 70. The apparatus of any of clauses 67 to 69, wherein the reporting request includes resource parameters identifying one or more resources from which the user equipment is to generate positioning data. Clause 71. The apparatus of clause 70, wherein the one or more resources include resources selected from the group including resources associated with an Assistance Data ID, resources associated with a Positioning Frequency Layer (PFL) ID, resources associated with a Transmit Receive (TRP) ID, resources associated with a Positioning Reference Signal (PRS) Set ID, a set of Positioning Reference Signal (PRS) resources, or a combination thereof. Clause 72. The apparatus of any of clauses 67-71, wherein the first positioning data is associated with a first subset of resources and the second positioning data is associated with a second subset of resources. Clause 73. The apparatus of clause 72, wherein the second subset of resources includes the first subset of resources. Clause 74. The apparatus of any of clauses 67 to 74, wherein the first positioning data and the second positioning data each include one or more elements associated with the same resource. Clause 75. An apparatus according to any one of clauses 67 to 75, wherein the first process is a trained machine learning process. Clause 76. An apparatus according to any one of clauses 67 to 75, wherein the second process is a trained machine learning process. Clause 77. An apparatus described in any of clauses 67 to 76, wherein the report request includes model parameters, the model parameters identifying one or more processes to monitor, the one or more processes including a first process and a second process. Clause 78. A non-transitory machine-readable storage medium storing instructions that, when executed by at least one processor of a server, cause the at least one processor to: sending a request for reporting positioning data to a user equipment; receiving report data indicative of a comparison between the first positioning data generated from the first process and the second positioning data generated from the second process; A non-transitory machine-readable storage medium that performs instructions, including generating and transmitting to the user equipment instructions that cause the user equipment to perform a first process based on the report data. Article 79. receiving third positioning data from the user equipment; The non-transitory machine-readable storage medium of clause 78, further comprising: generating and transmitting to the user equipment second instructions, based on the third positioning data, the second instructions causing the user equipment to perform a second process. Clause 80. A non-transitory machine-readable storage medium according to any one of clauses 78 to 79, wherein the report request includes timing parameters specifying a time interval for the user equipment to operate in the first mode. Clause 81. A non-transitory machine-readable storage medium according to any of clauses 78 to 80, wherein the reporting request includes resource parameters that identify one or more resources from which the user equipment will generate positioning data. Clause 82. The non-transitory machine-readable storage medium of clause 81, wherein the one or more resources include resources selected from the group including resources associated with an Assistance Data ID, resources associated with a Positioning Frequency Layer (PFL) ID, resources associated with a Transmit Receive (TRP) ID, resources associated with a Positioning Reference Signal (PRS) Set ID, a set of Positioning Reference Signal (PRS) resources, or a combination thereof. Clause 83. A non-transitory machine-readable storage medium according to any of clauses 78 to 82, wherein the first positioning data is associated with a first subset of resources and the second positioning data is associated with a second subset of resources. Clause 84. The non-transitory machine-readable storage medium of clause 83, wherein the second subset of resources includes the first subset of resources. Clause 85. A non-transitory machine-readable storage medium according to any one of clauses 78 to 84, wherein the first positioning data and the second positioning data each include one or more elements associated with the same resource. Clause 86. The non-transitory machine-readable storage medium of any of clauses 78 to 85, wherein the first process is a trained machine learning process. Clause 87. The non-transitory machine-readable storage medium of any of clauses 78 to 86, wherein the second process is a trained machine learning process. Clause 88. A non-transitory machine-readable storage medium according to any of clauses 78 to 87, wherein the report request includes model parameters, the model parameters identifying one or more processes to monitor, the one or more processes including a first process and a second process. Article 89. sending a request for reporting positioning data to a user equipment; receiving report data indicative of a comparison between the first positioning data generated from the first process and the second positioning data generated from the second process; A computer-implemented method comprising: generating, based on the reporting data, instructions to cause a user equipment to perform a first process; and transmitting the instructions to the user equipment. Article 90. receiving third positioning data from the user equipment; 90. The computer-implemented method of claim 89, further comprising: generating and transmitting to the user equipment second instructions, based on the third positioning data, that cause the user equipment to perform a second process. Clause 91. The computer-implemented method of any of clauses 89-90, wherein the report request includes timing parameters specifying a time interval for the user equipment to operate in the first mode. Clause 92. The computer-implemented method of any of clauses 89-91, wherein the reporting request includes resource parameters that identify one or more resources from which the user equipment is to generate positioning data. Clause 93. The computer-implemented method of clause 92, wherein the one or more resources include resources selected from the group including resources associated with an Assistance Data ID, resources associated with a Positioning Frequency Layer (PFL) ID, resources associated with a Transmit Receive (TRP) ID, resources associated with a Positioning Reference Signal (PRS) Set ID, a set of Positioning Reference Signal (PRS) resources, or a combination thereof. Clause 94. The computer-implemented method of any of clauses 89-93, wherein the first positioning data is associated with a first subset of resources and the second positioning data is associated with a second subset of resources. Clause 95. The computer-implemented method of clause 94, wherein the second subset of resources includes the first subset of resources. Clause 95. The computer-implemented method of any of clauses 89-95, wherein the first positioning data and the second positioning data each include one or more elements associated with the same resource. Clause 96. The computer-implemented method of any of clauses 89 to 95, wherein the first process is a trained machine learning process. Clause 97. The computer-implemented method of any of clauses 89 to 96, wherein the second process is a trained machine learning process. Clause 98. The computer-implemented method of any of clauses 89-97, wherein the report request includes model parameters, the model parameters identifying one or more processes to monitor, the one or more processes including a first process and a second process. Article 99. means for transmitting a request for reporting positioning data to a user equipment; means for receiving report data indicative of a comparison between the first positioning data generated from the first process and the second positioning data generated from the second process; and means for generating and transmitting to the user equipment instructions that cause the user equipment to perform a first process based on the reporting data. Article 100. means for receiving third positioning data from the user equipment; The positioning computing device described in clause 99, further comprising means for generating and transmitting to the user equipment second instructions that cause the user equipment to perform a second process based on the third positioning data. Clause 101. A positioning computing device as described in any of clauses 99 to 100, wherein the report request includes timing parameters specifying a time interval for the user equipment to operate in the first mode. Clause 102. A positioning computing device according to any of clauses 99 to 101, wherein the report request includes resource parameters identifying one or more resources from which the user equipment will generate positioning data. Clause 103. The positioning computing device of clause 102, wherein the one or more resources include resources selected from the group including resources associated with an assistance data ID, resources associated with a positioning frequency layer (PFL) ID, resources associated with a transmit receive (TRP) ID, resources associated with a positioning reference signal (PRS) set ID, a set of positioning reference signal (PRS) resources, or a combination thereof. Clause 104. A positioning computing device according to any of clauses 99 to 103, wherein the first positioning data is associated with a first subset of resources and the second positioning data is associated with a second subset of resources. Clause 105. The positioning computing device of clause 104, wherein the second subset of resources includes the first subset of resources. Clause 106. A positioning computing device according to any of clauses 99 to 105, wherein the first positioning data and the second positioning data each include one or more elements associated with the same resource. Clause 107. A positioning computing device according to any of clauses 99 to 105, wherein the first process is a trained machine learning process. Clause 108. A positioning computing device according to any of clauses 99 to 106, wherein the second process is a trained machine learning process. Clause 109. A positioning computing device as described in any of clauses 99 to 108, wherein the report request includes model parameters, the model parameters identifying one or more processes to monitor, the one or more processes including a first process and a second process.

[0088] C. Exemplary Hardware and Software Implementations Embodiments of the subject matter and functional operations described in this disclosure may be implemented in digital electronic circuitry, in tangibly embodied computer software or firmware, in computer hardware, including the structures disclosed herein and their structural equivalents, or in one or more combinations thereof. Embodiments of the subject matter described in this disclosure, including the user equipment (UE) engine 208A, application 212B, application programming interface (API) 402, process module 404, analysis module 406, notification module 408, API 502, notification engine 520, and API 602, may be implemented as one or more computer programs, i.e., as one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier, for execution by or to control the operation of a data processing device (or computing system). Additionally or alternatively, the program instructions may be encoded on an artificially generated propagated signal, such as a machine-generated electrical, optical, or electromagnetic signal, generated to encode information for transmission to a suitable receiver device for execution by the data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0089] The terms "apparatus," "device," and "system" refer to data processing hardware and encompass all kinds of apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. An apparatus, device, or system may also be or include special-purpose logic circuitry, such as an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). An apparatus, device, or system may optionally include, in addition to hardware, code that establishes an execution environment for a computer program, such as code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.

[0090] A computer program, which may also be referred to or described as a program, software, software application, application program, engine, module, software module, script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored within a portion of a file that holds other programs or data; for example, one or more scripts are stored within a markup language document, within a single file dedicated to the program in question, or within multiple cooperating files, such as files that store one or more modules, subprograms, or portions of code. A computer program can be deployed to run on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communications network.

[0091] The processes and logic flows described herein may be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by, and apparatus may also be implemented as, special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).

[0092] Computers suitable for running computer programs include, by way of example, general-purpose or special-purpose microprocessors, or both, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory or a random-access memory, or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or will be operatively coupled to one or more mass storage devices to receive data from, transfer data to, or both. However, a computer need not have such devices. Additionally, a computer may be incorporated into another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) or assisted Global Positioning System (AGPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0093] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0094] To provide for user interaction, embodiments of the subject matter described herein may be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, such as a mouse or trackball, by which the user can provide input to the computer. Other types of devices may be used to provide for user interaction as well; for example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user may be received in any form, including acoustic input, voice input, or tactile input. Additionally, the computer may interact with the user by sending documents to and receiving documents from a device used by the user, e.g., by sending a web page to a web browser on the user's device in response to a request received from the web browser.

[0095] An implementation of the subject matter described herein may be implemented in a computing system that includes back-end components such as a data server, or middleware components such as an application server, or front-end components such as a client computer having a graphical user interface or web browser through which a user can interact with an implementation of the subject matter described herein, or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication, such as a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs) such as the Internet.

[0096] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some implementations, a server sends data, such as HTML pages, to a user device, for example, for the purpose of displaying the data to and receiving user input from a user interacting with the user device acting as a client. Data generated at the user device, such as results of user interactions, may be received from the user device at the server.

[0097] While this specification contains many details, these details should not be construed as limitations on the scope of the disclosure or what may be claimed, but rather as descriptions of features specific to particular embodiments of the disclosure. Some features described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as functioning in a particular combination and are initially claimed as such, one or more features from a claimed combination may, in some cases, be deleted from the combination, and a claimed combination may be directed to a subcombination or a variation of a subcombination.

[0098] Similarly, while operations are illustrated in the figures in a particular order, this should not be understood as requiring such operations to be performed in the particular order or sequential order shown, or that all of the operations shown be performed, to achieve desirable results. In some situations, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged in multiple software products.

[0099] In each case where an HTML file is mentioned, other file types or formats may be substituted. For example, an HTML file may be replaced by an XML, JSON, plain text, or other type of file. Additionally, where a table or hash table is mentioned, other data structures (such as a spreadsheet, relational database, or structured file) may be used.

[0100] Although various embodiments have been described in detail herein with reference to the accompanying drawings, it will be apparent, however, that various modifications and changes may be made to those embodiments and that additional embodiments may be implemented without departing from the broader scope of the disclosed embodiments as set forth in the appended claims.

[0101] Furthermore, unless expressly defined otherwise herein, all terms are to be given their broadest possible interpretation, including the meaning indicated by the specification, as well as the meaning understood by a person skilled in the art and / or as defined in dictionaries, specialized books, etc. It should also be noted that as used in this specification and the appended claims, the singular forms "a," "an," and "the" include plural referents unless otherwise specified, and that the terms "comprises" and / or "comprising," as used herein, specify the presence or addition of one or more other features, aspects, steps, operations, elements, components, and / or groups thereof. Furthermore, terms such as "couple," "coupled," "operatively coupled," "operatively connected," and the like should be broadly understood to refer to connecting devices or components together mechanically, electrically, by wire, wirelessly, or otherwise, such that the connection enables the associated devices or components to operate (e.g., communicate) with each other as intended by that relationship. In this disclosure, the use of "or" means "and / or" unless otherwise stated. Furthermore, the use of the term "including" and other forms, such as "includes" and "included," is not limiting. In addition, terms such as "element" or "component" encompass both elements and components that comprise one unit and elements and components that comprise two or more subunits, unless otherwise stated. In addition, the section headings used herein are for organizational purposes only and should not be construed as limiting the subject matter described.

[0102] The foregoing is provided for the purposes of illustrating, explaining, and describing embodiments of the present disclosure. Modifications and adaptations to the embodiments will be apparent to those skilled in the art and may be made without departing from the scope or spirit of the present disclosure.

Claims

1. a non-transitory machine-readable storage medium storing instructions; at least one processor coupled to the non-transitory machine-readable storage medium, wherein the at least one processor executes the instructions to: transmitting a report request regarding positioning data to a user equipment, and receiving the report request by the user equipment; performing a first process for generating first positioning data and a second process for generating second positioning data; and generating report data indicative of a comparison between the first positioning data and the second positioning data; receiving the report data from the user equipment; and, based on the reporting data, generate and send to the user equipment instructions that cause the user equipment to perform at least one of the first process or the second process.

2. The apparatus of claim 1 , wherein the report request includes a timing parameter specifying a time interval for the user equipment to operate in the first mode.

3. The apparatus of claim 1 , wherein the report request includes resource parameters that identify one or more resources from which the user equipment is to generate the positioning data.

4. 4. The apparatus of claim 3, wherein the one or more resources comprise resources selected from the group including resources associated with an Assistance Data ID, resources associated with a Positioning Frequency Layer (PFL) ID, resources associated with a Transmission Received Point (TRP) ID, resources associated with a Positioning Reference Signal (PRS) Set ID, a set of Positioning Reference Signal (PRS) resources, or a combination thereof.

5. The apparatus of claim 1 , wherein the first positioning data is associated with a first subset of resources and the second positioning data is associated with a second subset of resources.

6. The apparatus of claim 5 , wherein the second subset of resources includes the first subset of resources.

7. The apparatus of claim 1 , wherein the first positioning data and the second positioning data each include one or more elements associated with a same resource.

8. The report request is sent to the user equipment: generating a first timestamp associated with a first element of the one or more elements of the first positioning data, the first element corresponding to a second element, and a second timestamp associated with the second element of the one or more elements of the second positioning data; comparing the first timestamp with the second timestamp; determining that the first timestamp and the second timestamp are within a predetermined time interval; The apparatus of claim 7 , further comprising: generating the reporting data based on the determination.

9. The report request may further include a request to the user equipment: The apparatus of claim 8 , further comprising determining that the first timestamp and the second timestamp match.

10. The report request may further include a request to the user equipment:

2. The apparatus of claim 1, wherein the first process is utilized to generate one or more elements of the first positioning data and the second process is utilized to generate one or more elements of the second positioning data.

11. The report request may further include a request to the user equipment: comparing the first positioning data with the second positioning data; determining, based on the comparison, that a difference between a first element of the first positioning data and a corresponding first element of the second positioning data satisfies a difference threshold; The device of claim 10 , further comprising: causing the device to transmit the report data based on the determination that the difference satisfies the difference threshold.

12. the user equipment operates in a second mode and performs the first process to generate the first positioning data before receiving the report request from the device, and the receipt of the report request by the user equipment causes the user equipment to: Switching from operation in the second mode to operation in the first mode; determining that at least the first element of the first positioning data and the corresponding first element of the second positioning data are within a predetermined time margin; comparing at least the first element of the first positioning data with the corresponding first element of the second positioning data; The apparatus of claim 11 , wherein the apparatus switches to operate in a third mode to generate additional positioning data in the condition where the comparison is greater than a predetermined threshold.

13. The apparatus of claim 12 , wherein the user equipment performs the second process to generate the additional positioning data while the user equipment is operating in the third mode.

14. The apparatus of claim 12 , wherein the user equipment performs a third process to generate the additional positioning data while the user equipment is operating in the third mode.

15. The apparatus of claim 1 , wherein the first process is a trained machine learning process.

16. The apparatus of claim 15 , wherein the second process is a trained machine learning process.

17. 2. The apparatus of claim 1, wherein the report request includes model parameters, the model parameters identifying one or more processes to monitor, the one or more processes including the first process and the second process.

18. 1. A non-transitory machine-readable storage medium storing instructions that, when executed by at least one processor of a location server, cause the at least one processor to: transmitting a report request regarding positioning data to a user equipment, wherein receiving the report request by the user equipment comprises: performing a first process for generating first positioning data and a second process for generating second positioning data; and transmitting, causing the user equipment to perform operations in a first mode, including generating report data indicative of a comparison between the first positioning data and the second positioning data; receiving the report data from the user equipment; and generating and transmitting to the user equipment instructions that cause the user equipment to perform at least one of the first process and / or the second process based on the report data.

19. 1. A computer-implemented method, the method comprising: transmitting, by a processor of the location server, a report request for positioning data to a user equipment, wherein receiving the report request by the user equipment includes: performing a first process for generating first positioning data and a second process for generating second positioning data; and transmitting, causing the user equipment to perform operations in a first mode, including generating report data indicative of a comparison between the first positioning data and the second positioning data; receiving, by the processor, the report data from the user equipment; generating and transmitting to the user equipment instructions that cause the user equipment to perform at least one of the first process and / or the second process based on the report data.

20. 20. The computer-implemented method of claim 19, wherein the report request includes timing parameters specifying a time interval for the user equipment to operate in the first mode.

21. 20. The computer-implemented method of claim 19, wherein the report request includes resource parameters that identify one or more resources from which the user equipment is to generate positioning data.

22. 22. The computer-implemented method of claim 21, wherein the one or more resources comprise resources selected from the group including resources associated with aiding, resources associated with a positioning frequency layer (PFL), resources associated with a transmit receive (TRP) ID, resources associated with a positioning reference signal (PRS) resource, or a combination thereof.

23. 20. The computer-implemented method of claim 19, wherein the first positioning data is associated with a first subset of resources and the second positioning data is associated with a second subset of resources.

24. 24. The computer-implemented method of claim 23, wherein the second subset of resources comprises the first subset of resources.

25. The report request is sent to the user equipment: generating a first timestamp associated with a first element of the one or more elements of the first positioning data, the first element corresponding to a second element, and a second timestamp associated with the second element of the one or more elements of the second positioning data; comparing the first timestamp with the second timestamp; determining that the first timestamp and the second timestamp are within a predetermined time interval; 25. The computer-implemented method of claim 24, further comprising generating the report data based on the determination.

26. 26. The computer-implemented method of claim 25, wherein the report request further causes the user equipment to determine that the first timestamp and the second timestamp match.

27. 20. The computer-implemented method of claim 19, wherein the report request further causes the user equipment to generate, using the first process, one or more elements of the first positioning data and, simultaneously, generate, using the second process, one or more elements of the second positioning data.

28. The report request may further include a request to the user equipment: comparing the first positioning data with the second positioning data; determining, based on the comparison, that a difference between a first element of the first positioning data and a corresponding first element of the second positioning data satisfies a difference threshold; 28. The computer-implemented method of claim 27, causing the processor of the location server to transmit the reporting data based on the determination that the difference meets the difference threshold.

29. Prior to receiving the report request from the processor of the location server, the user equipment operates in a second mode, and the receipt of the report request by the user equipment causes the user equipment to: Switching from operation in the second mode to operation in the first mode; determining that at least the first element of the first positioning data and the corresponding first element of the second positioning data are within a predetermined time margin; comparing at least the first element of the first positioning data with the corresponding first element of the second positioning data; 30. The computer-implemented method of claim 28, wherein for a condition in which the comparison is greater than a predetermined threshold, switching to operate in a third mode to generate additional positioning data.

30. 30. The computer-implemented method of claim 29, wherein the user equipment performs the second process to generate the additional positioning data while the user equipment is operating in the third mode.

31. 30. The computer-implemented method of claim 29, wherein while the user equipment is operating in the third mode, the user equipment performs a third process to generate the additional positioning data.

32. a non-transitory machine-readable storage medium storing instructions; at least one processor coupled to the non-transitory machine-readable storage medium, wherein the at least one processor executes the instructions to: Sending a request for reporting positioning data to the user equipment; receiving report data indicative of a comparison between the first positioning data generated from the first process and the second positioning data generated from the second process; and generating and transmitting to the user equipment, instructions that cause the user equipment to perform the first process based on the reporting data.

33. The at least one processor executes the instructions and further: receiving third positioning data from the user equipment; 33. The apparatus of claim 32, configured to generate and send to the user equipment second instructions based on the third positioning data to cause the user equipment to perform the second process.

34. sending a request for reporting positioning data to a user equipment; receiving report data indicative of a comparison between the first positioning data generated from the first process and the second positioning data generated from the second process; generating and transmitting instructions to the user equipment based on the report data to cause the user equipment to perform the first process.

35. receiving third positioning data from the user equipment; 35. The computer-implemented method of claim 34, further comprising: generating and transmitting to the user equipment second instructions based on the third positioning data to cause the user equipment to perform the second process.

Images