Cognitive environment detection in communication networks

Abstract

A computer-implemented method for detecting user environments in one or more communication networks is provided The method includes collecting, from one or more crowd data sources, unlabeled data to be used for user environment detection. The method includes filtering the collected data based on a plurality of features selected for user environment detection. The method includes labeling the filtered data as one of a plurality of labels, each label characterizing a user environment. The method includes classifying the labeled data, using one of a plurality of machine learning (ML) models to infer the user environment, as one of a plurality of classes, each class characterizing the user environment. The method includes using the classified data to detect user environments in the one or more communication networks.

Description

COGNITIVE ENVIRONMENT DETECTION IN COMMUNICATION NETWORKSTECHNICAL FIELD

[0001] Disclosed are embodiments related to user environment detection and, in particular, detecting user environments in one or more communication networks.BACKGROUND

[0002] User environment detection has been tackled in many previous research studies. There are traditional rule-based methods based on signal levels and, in other studies, methods that rely on machine learning (ML) to detect the environment.

[0003] Most of the studies define the problem as a binary classification one. They classify the environment as indoor or outdoor, and build a model to predict such labels. In order to collect data for the ML models, most of the studies rely on field tests from specific indoor and outdoor locations (which provide reliable ground truth information), while some others use custom-application data, call traces data (CTR), or crowd-sourced data.

[0004] For the data collection based on field test and custom application, the location of the samples is known, so it is automatically labeled, as illustrated in references 1, 2, 3, and 4, identified in the table below.

[0005] Crowdsourced data and call traces are not labelled, so they are usually combined with other sources to provide a solid ground truth. This approach is used in multiple studies where they use a combination of field data, crowd sourced data, and manually labeled data. See, e.g., references 3, 5, and 6. In reference 2, data based on map layers is used to complement and label call trace information. In reference 4, call trace data is correlated with data collected from a custom application.

[0006] Supervised classification is then applied on the labeled data to predict the class(indoor / outdoor). Tree models, logistic regression and neural networks are used in most cases. However, in references 5 and 3, semi-supervised models are used to enhance the classifier with the available unlabeled data.SUMMARY

[0007] Identifying user environment is critical, for example, for network planning, spectrum strategy and efficient resource utilization. An accurate / reliable indoor-outdoorclassification can be used to customize network parameters and configurations, and thus enhance user experience.

[0008] In existing methods, user environment is detected based on drive test, walk test and CTR data with a limited amount of collected samples. For the majority of previous methods, these samples are manually labeled, which is a very costly and time-consuming activity. Many of the existing methods use a limited amount of data from small clusters or areas, which limits their scalability and generalization to bigger radio networks, different radio technologies, radio suppliers and radio environments. Also, some of them are detecting only two types of environments: indoor and outdoor, without considering other environment types like semi-indoor and semi-outdoor, that are relevant to characterize users in transit.

[0009] Embodiments disclosed herein provide for cognitive environment detection (CED). Some embodiments disclosed herein provide an Al-based vendor-agnostic solution to detect user environment based on crowd-sourced data. Some embodiments disclosed herein use advanced automatic labeling techniques to tag large amounts of unlabeled crowdsourced data of, for example, 3G, 4G, and 5G users. Using the novel methods disclosed herein, testing in a large complex cluster of nodes showed -90% accurate performance.

[0010] In order to provide for the use crowd-sourced data, several steps were implemented including the collection of data, creation of a labeling engine, identification of the most effective features for the detection task, and exploring multiple modeling algorithms to better suit these features and predictions.

[0011] Embodiments disclosed herein rely on crowd-sourced data instead of drive tests, walk tests and CTR data, which enables the utilization of hundreds of thousands of users environment samples. The CED method of embodiments disclosed herein use a one-of- a-kind and automatic data labeling engine which may be based on three machine learning (ML) techniques: the first technique that can be used is the satellite images approach, which uses a deep learning model trained on satellite images; the second technique that can be used is the supervised thresholds approach, which uses GPS location speed, accuracy, and location features from map APIs; and the third technique that can be used is the majority voting approach, which weights the results of the first two approaches.

[0012] The CED method of embodiments disclosed herein differs from existing solutions in several ways, including:• Scalability: The CED method of embodiments disclosed herein can easily scale to different environments and bigger networks as it is trained on large samples from different areas and with minimal labeling effort;• Generalization: The CED method of embodiments disclosed herein can be generalized to different radio technologies and suppliers as they are all captured in crowd-sourced data, and can also detect more than two classes (Indoor, Outdoor, Semi-Indoor, Semi- Outdoor); and• Ground-truth data: The CED method of embodiments disclosed herein solves the limited ground-truth data problem using its automatic labeling engine, which can label thousands of samples.

[0013] An advantage of some of the embodiments disclosed herein is that they can help in efficient spectrum utilization and capacity planning based on traffic distribution for different environments, and can also improve the handling of network issues.

[0014] A further advantage of some of the embodiments disclosed herein is that they rely on the usage of crowd source data, which will reduce the dependency on the limited data of drive and walk tests. Most of the existing solutions have limitations in the availability of large amounts of data for model training. Embodiments disclosed herein can use hundreds of thousands of crowd data samples and reach to high accuracy.

[0015] A further advantage of some of the embodiments disclosed herein is that they have an edge in data labeling as they use new approaches for automatic labeling of the data.

[0016] A further advantage of some of the embodiments disclosed herein is that they can be used to target new solutions, improve customer experience and to enhance complaints handling. Using the methods of the embodiments disclosed herein will help in getting more insights for enhancing optimization services and issue handling since they can classify up to four categories (Indoor, Outdoor, Semi-Indoor, Semi -Outdoor).

[0017] A further advantage of some of the embodiments disclosed herein and the models they use is that they are easy to scale to different multi-vendor networks using existing crowd data without additional cost for the different technologies (3G, 4G, and 5G).

[0018] Advantages of the methods for detecting user environments in one or more communication networks of the embodiments disclosed herein compared with existing solutions include the following:

[0019] According to a first aspect, a computer-implemented method for detecting user environments in one or more communication networks is provided. The method includes collecting, from one or more crowd data sources, unlabeled data to be used for user environment detection. The method includes filtering the collected data based on a plurality of features selected for user environment detection. The method includes labeling the filtered data as one of a plurality of labels, each label characterizing a user environment. The method includes classifying the labeled data, using one of a plurality of machine learning (ML) models to infer the user environment, as one of a plurality of classes, each class characterizing the user environment. The method includes using the classified data to detect user environments in the one or more communication networks.

[0020] In some embodiments, the one or more communication networks include a plurality of user equipment (UEs) and the crowd data sources from which the data is collected include data from the plurality of UEs.

[0021] In some embodiments, the plurality of features selected for filtering relate to one or more of: user behavior; coverage; quality; and user specific location.

[0022] In some embodiments, the plurality of features selected for filtering comprises key performance indicators (KPIs) including one or more of: reference signal received power (RSRP); reference signal received quality (RSRQ); channel quality indicator (CQI); reference signal signal to noise ratio (RSSNR); timing advance (TA); location speed; and location accuracy.

[0023] In some embodiments, the plurality of labels, each label characterizing a user environment, includes one of: indoor or outdoor for two classes; indoor, outdoor or inbetween for three classes; and indoor, outdoor, semi-indoor, or semi -outdoor for four classes.

[0024] In some embodiments, the filtered data is labeled using a combination of approaches including a rule-based approach, a satellite images approach, and a majority voting approach.

[0025] In some embodiments, the rule-based approach comprises: applying data labeling rules to the filtered data based on: user speed information; user location information; and user environment information; and labelling the filtered data, based on applying the data labelling rules, as one of the plurality of labels to generate a rule-based approach labelled data set.

[0026] In some embodiments, the plurality of labels, each label characterizing a user environment, includes one of: indoor or outdoor for two classes; indoor, outdoor or inbetween for three classes; and indoor, outdoor, semi-indoor, or semi -outdoor for four classes.

[0027] In some embodiments, the user speed information includes user mobility speed; the user location information includes global positioning system (GPS) location accuracy; and the user environment information includes information provided by Mapbox TileQuery application programming interface (API).

[0028] In some embodiments, the satellite images approach comprises: retrieving a plurality of satellite images corresponding to a plurality of location points from the filtered data; scraping, from the retrieved plurality of satellite images, satellite images with fixed zoom; labeling the scraped satellite images, to generate a labeled data sample set, as one of the plurality of labels, each label characterizing a user environment, including one of: indoor or outdoor for two classes; indoor, outdoor or in-between for three classes; andindoor, outdoor, semi-indoor, or semi -outdoor for four classes; training, using the labeled data sample set, one or more ML models to infer the user environment as one of the plurality of labels; and labelling the filtered data, using the trained one or more ML models, as one of the plurality of labels to generate a satellite images approach labelled data set.

[0029] In some embodiments, the one or more ML models includes a pre-trained convolutional neural network (CNN) classifier. In some embodiments, the pre-trained CNN classifier is a ResNet-50 model.

[0030] In some embodiments, the satellite images approach further comprises changing one or more labels for the labelled filtered data based on location speed.

[0031] In some embodiments, the majority voting approach comprises: comparing the rule-based approach labelled data set with the satellite images approach labelled data set to determine the labels to be used for the filtered data, wherein, for each label: if the label for the filtered data in the rule-based approach labelled data set is the same as the label in the satellite images approach labelled data set, then the label is used; if the label for the filtered data in the rule-based approach labelled data set is different from the label in the satellite images approach labelled data set, then the label that is most frequent from the rule-based approach labelled data set and the satellite images approach labelled data set is used; and if the label for the filtered data in the rule-based approach labelled data set is different from the label in the satellite images approach labelled data set and there is no label that is most frequent, then a weighting is applied and the label with the highest weight is used.

[0032] In some embodiments, classifying the labeled data, using one of a plurality of machine learning (ML) models to infer the user environment, includes classifying the labelled data as one of: indoor or outdoor for two classes; indoor, outdoor or in-between for three classes; and indoor, outdoor, semi-indoor, or semi-outdoor for four classes.

[0033] In some embodiments, the one or more communication networks are one or more radio access networks (RANs). In some embodiments, using the classified data to detect user environments in the one or more communication networks includes using the classified data for one or more of: network planning; spectrum strategy; efficient resource utilization; customizing network parameters; customizing network configurations;enhancing user experience; capacity planning; virtual drive testing; performance diagnostics; and performance optimization.

[0034] According to a second aspect, a computing device is provided. The computing device comprises processing circuitry and a memory containing instructions executable by the processing circuitry for detecting user environments in one or more communication networks. The computing device is operative to collect, from one or more crowd data sources, unlabeled data to be used for user environment detection. The computing device is operative to filter the collected data based on a plurality of features selected for user environment detection. The computing device is operative to label the filtered data as one of a plurality of labels, each label characterizing a user environment. The computing device is operative to classify the labeled data, using one of a plurality of machine learning (ML) models to infer the user environment, as one of a plurality of classes, each class characterizing the user environment. The computing device is operative to use the classified data to detect user environments in the one or more communication networks.

[0035] According to a third aspect, a computer program is provided comprising instructions which, when executed by processing circuitry, causes the processing circuitry to perform the methods of any one of the embodiments of the first aspect.

[0036] According to a fourth aspect, a carrier is provided containing the computer program of the third aspect, wherein the carrier comprises one of an electronic signals, optical signal, radio signal or computer readable storage medium.

[0037] According to a fifth aspect, an apparatus is provided. The apparatus includes a memory and processing circuitry coupled to the memory, wherein the apparatus is configured to perform the methods of any one of the embodiments of the first aspect.BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate various embodiments.

[0039] FIG. l is a block diagram illustrating an architecture and a process for detecting user environments in one or more communication networks according to some embodiments.

[0040] FIG. 2 is a block diagram illustrating an architecture and a process for detecting user environments in one or more communication networks according to some embodiments.

[0041] FIG. 3 is a flow chart illustrating a process according to some embodiments.

[0042] FIG. 4 is a block diagram illustrating an architecture and a process for a labeling engine according to some embodiments.

[0043] FIG. 5 is a flow chart illustrating a process for a rule-based approach according to some embodiments.

[0044] FIG. 6 is a flow chart illustrating a process for a satellite images approach according to some embodiments.

[0045] FIG. 7 is a series of scraped and unlabeled satellite images according to some embodiments.

[0046] FIG. 8A is a pie chart illustrating the distribution of classes after applying a binary class model rule-based approach according to some embodiments.

[0047] FIG. 8B is a graph illustrating the RSRP distribution for 2 classes after applying a binary class model rule-based approach according to some embodiments

[0048] FIG. 9A is a pie chart illustrating the distribution of classes after applying a multi-class model rule-based approach according to some embodiments.

[0049] FIG. 9B is a graph illustrating the RSRP distribution for 3 classes after applying a multi-class model rule-based approach according to some embodiments.

[0050] FIG. 10A is a pie chart illustrating the distribution of classes after applying a multi-class model rule-based approach according to some embodiments.

[0051] FIG. 10B is a graph illustrating the RSRP distribution for 4 classes after applying a multi-class model rule-based approach according to some embodiments.

[0052] FIG. 11 A is a pie chart illustrating the distribution of classes after applying a binary class model satellite images approach according to some embodiments.

[0053] FIG. 1 IB is a graph illustrating the RSRP distribution for 2 classes after applying a binary class model satellite images approach according to some embodiments

[0054] FIG. 12A is a pie chart illustrating the distribution of classes after applying a multi-class model satellite images approach according to some embodiments.

[0055] FIG. 12B is a graph illustrating the RSRP distribution for 3 classes after applying a multi-class model satellite images approach according to some embodiments.

[0056] FIG. 13 A is a pie chart illustrating the distribution of classes after applying a multi-class model satellite images approach according to some embodiments.

[0057] FIG. 13B is a graph illustrating the RSRP distribution for 4 classes after applying a multi-class model satellite images approach according to some embodiments.

[0058] FIG. 14 is a flow chart illustrating a process for a majority voting approach according to some embodiments.

[0059] FIG. 15A is a pie chart illustrating the distribution of classes after applying a binary class model majority voting approach according to some embodiments.

[0060] FIG. 15B is a graph illustrating the RSRP distribution for 2 classes after applying a binary class model majority voting approach according to some embodiments.

[0061] FIG. 16A is a pie chart illustrating the distribution of classes after applying a multi-class model majority voting approach according to some embodiments.

[0062] FIG. 16B is a graph illustrating the RSRP distribution for 3 classes after applying a multi-class model majority voting approach according to some embodiments.

[0063] FIG. 17A is a pie chart illustrating the distribution of classes after applying a multi-class model majority voting approach according to some embodiments.

[0064] FIG. 17B is a graph illustrating the RSRP distribution for 4 classes after applying a multi-class model majority voting approach according to some embodiments.

[0065] FIG. 18 is a block diagram of an apparatus according to some embodiments.DETAILED DESCRIPTION

[0066] FIG. l is a block diagram illustrating an architecture and a process for detecting user environments in one or more communication networks according to some embodiments. Referring now to FIG. 1, block diagram 100 includes a data collection and feature selection module 110, a labeling engine module 120, and a modeling module 130.In some embodiments, data collection and feature selection module 110 is configured to collect user environment KPIs from crowd data sources and filter the most important features. In some embodiments, labeling engine module 120 is configured to apply different labeling approaches to divide data into different classes. In some embodiments, modeling module 130 is configured to apply different tree-based and deep learning models. This architecture and process of the embodiments disclosed herein provides a vendoragnostic multiclass ML model using crowd-sourced data that identifies the user environment without relying on RAN KPIs.

[0067] Data Collection and Feature Selection

[0068] To detect the user environment, we focus on some features which can capture the user behavior like, for example, coverage, quality, and user specific location. Crowdsourced data offers millions of samples collected directly from user devices / equipments - i.e., user centric - and covers a variety of radio metrics which can be used for environment detection purpose. It also includes multiple technologies and vendors.

[0069] In some embodiments, crowd sourced data is utilized by, for example, collecting 250k points that contain valid measurement data for 3 Ik unique location points which are unlabeled and includ these features: RSRP, RSRQ, CQI, RSSNR, Timing Advance, Location Speed and Location Accuracy. These features are considered as part of identifying indoor and outdoor environments based on the subsequent feature analysis for the diclosed approaches.

[0070] FIG. 2 is a block diagram illustrating an architecture and a process for detecting user environments in one or more communication networks according to some embodiments. Referring to FIG. 2, the architecture and process 200 shows the flow beginning with the collection of crowd sourced data 210. The flow continues with feature selection 220 and model selection 230. In exemplary embodiments, the labeling model will get the important features coming from crowd sourced data directly from the UEs.

[0071] Continuing with the flow shown in FIG. 2, for model selection 230, three model options can be chosen depending on the use-case - i.e., either two, three or four classes depending on the resolution needed. The first option is binary classifier 240 (2 classes), the second option is 3 class multi classifier 250 (3 classes), and the third option is 4 class multi classifier 260 (4 classes). After the trained model is applied, at modelinference 270, each data point will be classified accordingly, at I / O labeled crowd sourced data 280. The classified data can then be used in relevant use-cases, at I / O dependent use cases 290.

[0072] FIG. 3 is a flow chart illustrating a process according to some embodiments. Process 300 is a computer-implemented method for detecting user environments in one or more communication networks is provided. Referring now to FIG. 3, process 300 may begin with step s302.

[0073] Step s302 comprises collecting, from one or more crowd data sources, unlabeled data to be used for user environment detection.

[0074] Step s304 comprises filtering the collected data based on a plurality of features selected for user environment detection.

[0075] Step s306 comprises labeling the filtered data as one of a plurality of labels, each label characterizing a user environment.

[0076] Step s308 comprises classifying the labeled data, using one of a plurality of machine learning (ML) models to infer the user environment, as one of a plurality of classes, each class characterizing the user environment.

[0077] Step s310 comprises using the classified data to detect user environments in the one or more communication networks.

[0078] In some embodiments, the one or more communication networks include a plurality of user equipment (UEs) and the crowd data sources from which the data is collected include data from the plurality of UEs.

[0079] In some embodiments, the plurality of features selected for filtering relate to one or more of: user behavior; coverage; quality; and user specific location.

[0080] In some embodiments, the plurality of features selected for filtering comprises key performance indicators (KPIs) including one or more of: reference signal received power (RSRP); reference signal received quality (RSRQ); channel quality indicator (CQI); reference signal signal to noise ratio (RSSNR); timing advance (TA); location speed; and location accuracy.

[0081] In some embodiments, the plurality of labels, each label characterizing a user environment, includes one of: indoor or outdoor for two classes; indoor, outdoor or in-between for three classes; and indoor, outdoor, semi-indoor, or semi -outdoor for four classes.

[0082] In some embodiments, the filtered data is labeled using a combination of approaches including a rule-based approach, a satellite images approach, and a majority voting approach.

[0083] In some embodiments, the rule-based approach comprises: applying data labeling rules to the filtered data based on: user speed information; user location information; and user environment information; and labelling the filtered data, based on applying the data labelling rules, as one of the plurality of labels to generate a rule-based approach labelled data set.

[0084] In some embodiments, the plurality of labels, each label characterizing a user environment, includes one of: indoor or outdoor for two classes; indoor, outdoor or inbetween for three classes; and indoor, outdoor, semi-indoor, or semi -outdoor for four classes.

[0085] In some embodiments, the user speed information includes user mobility speed; the user location information includes global positioning system (GPS) location accuracy; and the user environment information includes information provided by Mapbox TileQuery application programming interface (API).

[0086] In some embodiments, the satellite images approach comprises: retrieving a plurality of satellite images corresponding to a plurality of location points from the filtered data; scraping, from the retrieved plurality of satellite images, satellite images with fixed zoom; labeling the scraped satellite images, to generate a labeled data sample set, as one of the plurality of labels, each label characterizing a user environment, including one of: indoor or outdoor for two classes; indoor, outdoor or in-between for three classes; and indoor, outdoor, semi-indoor, or semi -outdoor for four classes; training, using the labeled data sample set, one or more ML models to infer the user environment as one of the plurality of labels; and labelling the filtered data, using the trained one or more ML models, as one of the plurality of labels to generate a satellite images approach labelled data set.

[0087] In some embodiments, the one or more ML models includes a pre-trained convolutional neural network (CNN) classifier. In some embodiments, the pre-trained CNN classifier is a ResNet-50 model.

[0088] In some embodiments, the satellite images approach further comprises changing one or more labels for the labelled filtered data based on location speed.

[0089] In some embodiments, the majority voting approach comprises: comparing the rule-based approach labelled data set with the satellite images approach labelled data set to determine the labels to be used for the filtered data, wherein, for each label: if the label for the filtered data in the rule-based approach labelled data set is the same as the label in the satellite images approach labelled data set, then the label is used; if the label for the filtered data in the rule-based approach labelled data set is different from the label in the satellite images approach labelled data set, then the label that is most frequent from the rule-based approach labelled data set and the satellite images approach labelled data set is used; and if the label for the filtered data in the rule-based approach labelled data set is different from the label in the satellite images approach labelled data set and there is no label that is most frequent, then a weighting is applied and the label with the highest weight is used.

[0090] In some embodiments, classifying the labeled data, using one of a plurality of machine learning (ML) models to infer the user environment, includes classifying the labelled data as one of: indoor or outdoor for two classes; indoor, outdoor or in-between for three classes; and indoor, outdoor, semi-indoor, or semi-outdoor for four classes.

[0091] In some embodiments, the one or more communication networks are one or more radio access networks (RANs). In some embodiments, using the classified data to detect user environments in the one or more communication networks includes using the classified data for one or more of: network planning; spectrum strategy; efficient resource utilization; customizing network parameters; customizing network configurations; enhancing user experience; capacity planning; virtual drive testing; performance diagnostics; and performance optimization.

[0092] Labeling Engine

[0093] Embodiments of the disclosed CED method use a novel automatic labeling engine to define the user environment based on, for example, the crowd sourced data KPIs (Indoor - Outdoor - In Between). Novel approaches were implemented to capture the KPIs importance and classify them from two to four classes depending on the radio and location properties.

[0094] Referring now to FIG. 2, the labeling engine functionality is part of Feature Selection 220 in FIG. 2. The labeling engine receives unlabeled Crowd Sourced Data 210, applies the combination of approaches to label the unlabeled crowd data for Model Selection 230, as one of a plurality of labels, each label characterizing a user environment: Binary Classifier 240 (2 classes - indoor or outdoor), 3 Class Multi-Classifier 250 (3 classes - indoor, outdoor or in-between), or 4 Class Multi-Classifier 260 (4 classes - indoor, outdoor, semiindoor, or semi-outdoor).

[0095] FIG. 4 is a block diagram illustrating an architecture and a process 400 for a labeling engine 410 according to some embodiments. Referring now to FIG. 4, the labelling engine 410 uses a combination of approaches including a rule-based approach 420, a satellite images approach 430, and a majority voting approach 440. The labeling engine 410 receives unlabeled crowd data 450 and applies the combination of approaches to label the unlabeled crowd data as one of a plurality of labels, each label characterizing a user environment:• 2 classes - indoor or outdoor 460;• 3 classes - indoor, outdoor or in-between 470; or• 4 classes - indoor, outdoor, semi-indoor, or semi-outdoor 480.

[0096] Rule-Based Approach

[0097] The Rule-Based Approach 420 metrics include user mobility speed, GPS location accuracy, and features from Mapbox TileQuery API. The crowd sourced data provides multiple KPIs for environment detection. To begin with, a simple rule-based approach can be used to label the data. The approach mainly relies on user speed and location accuracy along with location features captured from Mapbox TileQuery API, which gives an indication of the environment type. The TileQuery API represents a pivotal component in modern geospatial data processing, providing an efficient solution for querying and retrieving map data. The API seamlessly integrates with mapping platforms, enabling an interaction with tile-based geospatial datasets. Leveraging this API helped to retrieve specific data tiles based on their geographical coordinates from crowd-sourced data. Each tile contains a wealth of information about a specific geographic area like, for example, Geometric Information, Topography, Elevation data, Land Use and Land Cover, Transportation Networks, and Administrative data.

[0098] FIG. 5 is a flow chart illustrating an exemplary process for a rule-based approach according to some embodiments. This process 500 shows how the received unlabeled crowd source data can be used for designing the labeling rules. Referring now to FIG. 5, at 502, the received user mobility speed data is used, and if, for example, the speed is > 2 m / s, then, at 504, the Outdoor label is applied. If, for example, the speed is not > 2 m / s, then, at 506, the GPS location accuracy data is used, and if, for example, the accuracy is < 10m, then, at 508, the Outdoor label is applied. If the accuracy is, for example, not < 10m, then, at 510, the process continues with determining what label should be applied depending on whether it is determined that the data indicates majority indoor or not. If majority indoor, then, at 512, a determination is made whether an indoor place is, for example, within 10m. If yes, at 514, if the speed, for example, is < 1 m / s, then, at 516, the Indoor label is applied. If not, then at 518, the Semi-indoor label is applied. If the indoor place is not, for example, within 10 m, then, at 518, the Semi-indoor label is applied. If not majority indoor, then, at 520, a determination is made whether an outdoor place is, for example, within 10m. If yes, at 522, if, for example, the speed is > 1 m / s, then, at 524, the Outdoor label is applied. If not, then at 526, the Semi-outdoor label is applied. If the outdoor place is not, for example, within 10 m, the, at 526, the Semi -outdoor label is applied.

[0099] Satellite Images Approach

[0100] The satellite images approach 430 uses satellite images and a computer vision model. Satellite images, which can be accessed using only longitude and latitude, are continuously updated. These images are highly useful for identifying the environment type of each location. In comparison to the rule-based approach, which varies in accuracy for different locations on earth, satellite images are highly accurate and reliable. Thus, they are an excellent tool for conducting research in various fields that require precise location-based data.

[0101] The satellite images approach model pipeline can be illustrated as:

[0102] FIG. 6 is a flow chart illustrating a process 600 for a satellite images approach according to some embodiments. Referring now to FIG. 6, at 602, data selection of, for example, unannotated images used for the model occurs using, for example, Google Maps API. From the satellite images retrieved, at 604, chosen data points locations within range are selected based on the crowd-sourced data. At 606, scraping those points from the map occurs with fixed zoom and window. A series of exemplary scraped and unlabeled satellite images 700 according to some embodiments are shown in FIG. 7.

[0103] Referring back to FIG. 6, at 608, the scraped and unlabeled satellite images are labeled manually as a seed to a pre-trained CNN classifier. The manual labeling (Indoor - Outdoor) images are evaluated, at 610, to determine if there are sufficient labeled images. If not, then further manual labeling of images, at 608, continues. If there are sufficient labeled images, the process continues to 612, modeling.

[0104] Since labelling the entire dataset is time consuming, the manually labeled data sample is used, at 612, 614 and 616, to train a labelling model to classify the images. Using, for example, a pre-trained ResNet-50 model, at 614, multiple models were re-trained to detect (2 classes - 3 classes - 4 classes). If it is determined, at 616, that there are acceptable accuracies, the models are then applied, at 618, to get the labels for the entire crowd-sourced dataset. If it is determined that there are not acceptable accuracies, then further manual labeling of images, at 608, continues and 610 to 616 are repeated until there are acceptable accuracies.

[0105] Retraining a ResNet-50 model on a computer vision dataset involves leveraging transfer learning, a technique that harnesses the pre-trained knowledge of a model on a large dataset and adapts it to a specific task or dataset. Modifications were implemented to enable the ResNet 50 model used to adapt to the Indoor / Outdoor dataset that was created manually. The input is an image coming from the google maps API and its output is the user environment class (Indoor - Outdoor - Semi Indoor- Semi Outdoor).

[0106] The initial layers (25) of the ResNet-50 were frozen to retain the pre-trained features and avoid overfitting, especially when dealing with limited labeled data. These early layers serve as feature extractors, capturing low-level spatial information such as edges and textures. Freezing these foundational features is enables preserving learned representations, allowing the model to adapt its higher-order features to the specific characteristics of the input dataset.

[0107] Enhanced Satellite Images Approach

[0108] Upon conducting further labeling of the complete dataset using the retrained ResNet model, it is possible that the neural network model may have misclassified certain samples. To counter this, location speed was used to enhance the model outputs:• If the generated (label is indoor) and (speed > 1.5 m / s), then the label is changed to Outdoor.• If the generated (label is indoor) and (speed [0.5: 1.5] m / s), then the label is changed to In-Between or Semi-Indoor if the model is 3 classes - 4 classes respectively.

[0109] Referring to FIG. 6, after labelling the whole dataset using the model, at 620, enhanced labelling can be used and, at 622, it is determined whether an Indoor label is generated from the model. If it is, then, at 624, speed is evaluated to determine whether to force label the data as Semi-Indoor, at 626, or force label the data as Outdoor, at 630. If it is determined, at 622, that an Indoor label is not generated from the model, then enhanced labelling continues, at 620.

[0110] Majority Voting Approach

[0111] So far, the following approaches have been used for labeling: The Rule-based approach and two Satellite-images-based approaches. The majority voting approach 440 is a labeling technique that makes a vote between the previous techniques to decide the final label. This helps in reducing outliers in labeling and utilizes both the ML-based and rulebased approaches to get the best accuracy.

[0112] Before describing the details and application of the majority voting approach, we provide some exemplary classification results from using the rule-based labeling approach and the satellite images labeling approach for binary and multi-class models.

[0113] Binary-Class Model Rule-Based Approach

[0114] FIG. 8 A is a pie chart 810 illustrating the distribution of classes after applying a binary class model rule-based approach according to some embodiments. FIG. 8B is a graph 820 illustrating the RSRP distribution for 2 classes after applying a binary class model rule-based approach according to some embodiments.

[0115] Multi-Class Model Rule-Based Approach - 3 Classes

[0116] FIG. 9A is a pie chart 910 illustrating the distribution of classes after applying a multi-class model rule-based approach according to some embodiments. FIG. 9B is a graph 920 illustrating the RSRP distribution for 3 classes after applying a multi-class model rulebased approach according to some embodiments.

[0117] Multi-Class Model Rule-Based Approach - 4 Classes

[0118] FIG. 10A is a pie chart 1010 illustrating the distribution of classes after applying a multi-class model rule-based approach according to some embodiments. FIG. 10B is a graph 1020 illustrating the RSRP distribution for 4 classes after applying a multi-class model rule-based approach according to some embodiments.

[0119] Binary-Class Model Satellite Images Approach

[0120] FIG. 11 A is a pie chart 1110 illustrating the distribution of classes after applying a binary class model satellite images approach according to some embodiments. FIG. 1 IB is a graph 1120 illustrating the RSRP distribution for 2 classes after applying a binary class model satellite images approach according to some embodiments

[0121] Multi-Class Model Satellite Images Approach - 3 Classes

[0122] FIG. 12A is a pie chart 1210 illustrating the distribution of classes after applying a multi-class model satellite images approach according to some embodiments. FIG. 12B is a graph 1220 illustrating the RSRP distribution for 3 classes after applying a multi-class model satellite images approach according to some embodiments.

[0123] Multi-Class Model Satellite Images Approach - 4 Classes

[0124] FIG. 13 A is a pie chart 1310 illustrating the distribution of classes after applying a multi-class model satellite images approach according to some embodiments. FIG. 13B is a graph 1320 illustrating the RSRP distribution for 4 classes after applying a multi-class model satellite images approach according to some embodiments.

[0125] Turning now to the details and application of the majority voting approach, FIG. 14 is a flow chart illustrating a process 1400 for a majority voting approach according to some embodiments. At 1410, if the satellite image approach and enhanced satellite image approach produce the same label, then, at 1420, this label is used. However, if they produce different labels, then, at 1430, the most frequent label (mode) from all the labelling approaches (satellite and rule-based) is used, at 1440, to assign the labels. If there is nodominant label from all the labeling approaches, then their outcomes are weighted, at 1450, where higher weight is given to the satellite images approach as it has shown the best accuracy in the testing that has been conducted.

[0126] Binary Class Model

[0127] For the binary labeling (indoor - outdoor), applying the majority voting approach as described, the mode was used between outcomes of the labeling approaches. FIG. 15A is a pie chart 1510 illustrating the distribution of classes after applying a binary class model majority voting approach according to some embodiments. FIG. 15B is a graph 1520 illustrating the RSRP distribution for 2 classes after applying a binary class model majority voting approach according to some embodiments.

[0128] Multi-Class Models

[0129] For multi-class labeling (indoor - outdoor - in-between; indoor - outdoor - semi-indoor - semi-outdoor), the majority voting approach is applied according to the process of FIG. 14. FIG. 16A is a pie chart 1610 illustrating the distribution of classes after applying a multi-class model majority voting approach according to some embodiments. FIG. 16B is a graph 1620 illustrating the RSRP distribution for 3 classes after applying a multi-class model majority voting approach according to some embodiments. FIG. 17A is a pie chart 1710 illustrating the distribution of classes after applying a multi-class model majority voting approach according to some embodiments. FIG. 17B is a graph 1720 illustrating the RSRP distribution for 4 classes after applying a multi-class model majority voting approach according to some embodiments.

[0130] Classification Modeling

[0131] Referring now to FIG. 2, the labeling engine applies the combination of approaches to label the unlabeled crowd data for Model Selection 230, as one of a plurality of labels, each label characterizing a user environment: Binary Classifier 240 (2 classes - indoor or outdoor), 3 Class Multi-Classifier 250 (3 classes - indoor, outdoor or in-between), or 4 Class Multi-Classifier 260 (4 classes - indoor, outdoor, semi-indoor, or semi -outdoor). After labeling, these ML classification models 240, 250, and 260 are utilized for Model Inference 270 to effectively adapt to the employed features and detect the corresponding user environment. For each labeling approach, multiple models are created, and each produce adifferent number of classes - i.e., 2 classes, 3 classes, and 4 classes - to characterize the environment including multiclass labels as follows:

[0132] The usage of the three multi-class models depends on the use-case and the resolution it requires.

[0133] For the different number of classes generated by the labeling engine, the following models, for example, were used:• XGBoost• SVM Classifier• Extra Trees Classifier• Random Forest Classifier

[0134] The input for training these models is the raw crowd-sourced data features (RSRP, RSRQ, CQI, RS SNR, Timing Advance, Location Speed and Location Accuracy) and the labels generated from the labeling engine. Once the model is trained, it can predict the user environment for new input data.

[0135] The best machine learning algorithm for the different number of classes were Extra-Trees classifier and Random Forest classifier. Both algorithms build several decision trees and weight their outcomes to predict the correct label. They share similarities but differ in the way they build and utilize the individual decision trees within their ensembles.

[0136] Results

[0137] The following table shows the best for all labeling techniques using different number of classes:

[0138] The labelled input data was split into training and testing samples, where the training data was 80% of the samples, while the remaining 20% were used to test the performance of the models on unseen data. The training and testing accuracy were compared to ensure the consistency of the performance. The balanced accuracy is also calculated for the testing dataset to confirm that the model is not biased towards a certain output.

[0139] FIG. 18 is a block diagram of an apparatus 1800, according to some embodiments. As shown in FIG. 18, the apparatus may comprise: processing circuitry (PC) 1802, which may include one or more processors (P) 1855 (e.g., a general purpose microprocessor and / or one or more other processors, such as an application specific integrated circuit (ASIC), field-programmable gate arrays (FPGAs), and the like); a network interface 1848 comprising a transmitter (Tx) 1845 and a receiver (Rx) 1847 for enabling the apparatus to transmit data to and receive data from other computing devices connected to a network 1810 (e.g., an Internet Protocol (IP) network) to which network interface 1848 is connected; and a local storage unit (a.k.a., “data storage system”) 1808, which may include one or more non-volatile storage devices and / or one or more volatile storage devices. In embodiments where PC 1802 includes a programmable processor, a computer program product (CPP) 1841 may be provided. CPP 1841 includes a computer readable medium (CRM) 1842 storing a computer program (CP) 1843 comprising computer readable instructions (CRI) 1844. CRM 1842 may be a non-transitory computer readable medium, such as, magnetic media (e.g., a hard disk), optical media, memory devices (e.g., random access memory, flash memory), and the like.

[0140] In some embodiments, the CRI 1844 of computer program 1843 is configured such that when executed by PC 1802, the CRI 1844 causes the apparatus 1800 to performsteps / functions described herein (e.g., steps / functions described herein with reference to FIGS. 1-6 and 14). In other embodiments, the apparatus 1800 may be configured to perform steps / functions described herein without the need for code. That is, for example, PC 1802 may consist merely of one or more ASICs. Hence, the features of the embodiments described herein may be implemented in hardware and / or software.

[0141] While various embodiments of the present disclosure are described herein, it should be understood that they have been presented by way of example only, and not limitation. Thus, the breadth and scope of the present disclosure should not be limited by any of the above-described exemplary embodiments. Moreover, any combination of the abovedescribed elements in all possible variations thereof is encompassed by the disclosure unless otherwise indicated herein or otherwise clearly contradicted by context.

[0142] Additionally, while the processes described above and illustrated in the drawings are shown as a sequence of steps, this was done solely for the sake of illustration. Accordingly, it is contemplated that some steps may be added, some steps may be omitted, the order of the steps may be re-arranged, and some steps may be performed in parallel.

[0143] References:

[0144] Abbreviations:

Claims

CLAIMS:

1. A computer-implemented method for detecting user environments in one or more communication networks, comprising: collecting, from one or more crowd data sources, unlabeled data to be used for user environment detection; filtering the collected data based on a plurality of features selected for user environment detection; labeling the filtered data as one of a plurality of labels, each label characterizing a user environment; classifying the labeled data, using one of a plurality of machine learning (ML) models to infer the user environment, as one of a plurality of classes, each class characterizing the user environment; and using the classified data to detect user environments in the one or more communication networks.

2. The computer-implemented method according to claim 1, wherein the one or more communication networks include a plurality of user equipment (UEs) and the crowd data sources from which the data is collected include data from the plurality of UEs.

3. The computer-implemented method according to claims 1 or 2, wherein the plurality of features selected for filtering relate to one or more of: user behavior; coverage; quality; and user specific location.

4. The computer-implemented method according to any one of claims 1-3, wherein the plurality of features selected for filtering comprises key performance indicators (KPIs) including one or more of: reference signal received power (RSRP); reference signal received quality (RSRQ); channel quality indicator (CQI); reference signal signal to noise ratio (RSSNR);timing advance (TA); location speed; and location accuracy.

5. The computer-implemented method according to any one of claims 1-4, wherein the plurality of labels, each label characterizing a user environment, includes one of: indoor or outdoor for two classes; indoor, outdoor or in-between for three classes; and indoor, outdoor, semi-indoor, or semi-outdoor for four classes.

6. The computer-implemented method according to any one of claims 1-5, wherein the filtered data is labeled using a combination of approaches including a rule-based approach, a satellite images approach, and a majority voting approach.

7. The computer-implemented method according to claim 6, wherein the rulebased approach comprises: applying data labeling rules to the filtered data based on: user speed information; user location information; and user environment information; and labelling the filtered data, based on applying the data labelling rules, as one of the plurality of labels to generate a rule-based approach labelled data set.

8. The computer-implemented method according to claim 7, wherein the plurality of labels, each label characterizing a user environment, includes one of: indoor or outdoor for two classes; indoor, outdoor or in-between for three classes; and indoor, outdoor, semi-indoor, or semi-outdoor for four classes.

9. The computer-implemented method according to claim 7, wherein: the user speed information includes user mobility speed; the user location information includes global positioning system (GPS) location accuracy; andthe user environment information includes information provided by Mapbox TileQuery application programming interface (API).

10. The computer-implemented method according to claim 7, wherein the satellite images approach comprises: retrieving a plurality of satellite images corresponding to a plurality of location points from the filtered data; scraping, from the retrieved plurality of satellite images, satellite images with fixed zoom; labeling the scraped satellite images, to generate a labeled data sample set, as one of the plurality of labels, each label characterizing a user environment, including one of: indoor or outdoor for two classes; indoor, outdoor or in-between for three classes; and indoor, outdoor, semi-indoor, or semi-outdoor for four classes; training, using the labeled data sample set, one or more ML models to infer the user environment as one of the plurality of labels; and labelling the filtered data, using the trained one or more ML models, as one of the plurality of labels to generate a satellite images approach labelled data set.

11. The computer-implemented method according to claim 10, wherein the one or more ML models includes a pre-trained convolutional neural network (CNN) classifier.

12. The computer-implemented method according to claim 11, wherein the pretrained CNN classifier is a ResNet-50 model.

13. The computer-implemented method according to claim 10, wherein the satellite images approach further comprises changing one or more labels for the labelled filtered data based on location speed.

14. The computer-implemented method according to any one of claims 10-13, wherein the majority voting approach comprises: comparing the rule-based approach labelled data set with the satellite images approach labelled data set to determine the labels to be used for the filtered data, wherein, foreach label: if the label for the filtered data in the rule-based approach labelled data set is the same as the label in the satellite images approach labelled data set, then the label is used; if the label for the filtered data in the rule-based approach labelled data set is different from the label in the satellite images approach labelled data set, then the label that is most frequent from the rule-based approach labelled data set and the satellite images approach labelled data set is used; and if the label for the filtered data in the rule-based approach labelled data set is different from the label in the satellite images approach labelled data set and there is no label that is most frequent, then a weighting is applied and the label with the highest weight is used.

15. The computer-implemented method according to any one of claims 1-14, wherein classifying the labeled data, using one of a plurality of machine learning (ML) models to infer the user environment, includes classifying the labelled data as one of: indoor or outdoor for two classes; indoor, outdoor or in-between for three classes; and indoor, outdoor, semi-indoor, or semi-outdoor for four classes.

16. The computer-implemented method according to any one of claims 1-15, wherein the one or more communication networks are one or more radio access networks (RANs).

17. The computer-implemented method according to any one of claims 1-16, wherein using the classified data to detect user environments in the one or more communication networks includes using the classified data for one or more of: network planning; spectrum strategy; efficient resource utilization; customizing network parameters; customizing network configurations; enhancing user experience;capacity planning; virtual drive testing; performance diagnostics; and performance optimization.

18. A computing device comprising: processing circuitry; and a memory containing instructions executable by the processing circuitry for detecting user environments in one or more communication networks, the computing device operative to: collect, from one or more crowd data sources, unlabeled data to be used for user environment detection; filter the collected data based on a plurality of features selected for user environment detection; label the filtered data as one of a plurality of labels, each label characterizing a user environment; classify the labeled data, using one of a plurality of machine learning (ML) models to infer the user environment, as one of a plurality of classes, each class characterizing the user environment; and use the classified data to detect user environments in the one or more communication networks.

19. The computing device according to claim 18, wherein the one or more communication networks include a plurality of user equipment (UEs) and the crowd data sources from which the data is collected include data from the plurality of UEs.

20. The computing device according to claims 18 or 19, wherein the plurality of features selected for filtering relate to one or more of: user behavior; coverage; quality; and user specific location.

21. The computing device according to any one of claims 18-20, wherein the plurality of features selected for filtering comprises key performance indicators (KPIs) including one or more of: reference signal received power (RSRP); reference signal received quality (RSRQ); channel quality indicator (CQI); reference signal signal to noise ratio (RSSNR); timing advance (TA); location speed; and location accuracy.

22. The computing device according to any one of claims 18-21, wherein the plurality of labels, each label characterizing a user environment, includes one of: indoor or outdoor for two classes; indoor, outdoor or in-between for three classes; and indoor, outdoor, semi-indoor, or semi-outdoor for four classes.

23. The computing device according to any one of claims 18-22, wherein the filtered data is labeled using a combination of approaches including a rule-based approach, a satellite images approach, and a majority voting approach.

24. The computing device according to claim 23, wherein the rule-based approach comprises: applying data labeling rules to the filtered data based on: user speed information; user location information; and user environment information; and labelling the filtered data, based on applying the data labelling rules, as one of the plurality of labels to generate a rule-based approach labelled data set.

25. The computing device according to claim 24, wherein the plurality of labels, each label characterizing a user environment, includes one of: indoor or outdoor for two classes;indoor, outdoor or in-between for three classes; and indoor, outdoor, semi-indoor, or semi-outdoor for four classes.

26. The computing device according to claim 24, wherein: the user speed information includes user mobility speed; the user location information includes global positioning system (GPS) location accuracy; and the user environment information includes information provided by Mapbox TileQuery application programming interface (API).

27. The computing device according to claim 24, wherein the satellite images approach comprises: retrieving a plurality of satellite images corresponding to a plurality of location points from the filtered data; scraping, from the retrieved plurality of satellite images, satellite images with fixed zoom; labeling the scraped satellite images, to generate a labeled data sample set, as one of the plurality of labels, each label characterizing a user environment, including one of: indoor or outdoor for two classes; indoor, outdoor or in-between for three classes; and indoor, outdoor, semi-indoor, or semi-outdoor for four classes; training, using the labeled data sample set, one or more ML models to infer the user environment as one of the plurality of labels; and labelling the filtered data, using the trained one or more ML models, as one of the plurality of labels to generate a satellite images approach labelled data set.

28. The computing device according to claim 27, wherein the one or more ML models includes a pre-trained convolutional neural network (CNN) classifier.

29. The computing device according to claim 28, wherein the pre-trained CNN classifier is a ResNet-50 model.

30. The computing device according to claim 27, wherein the satellite imagesapproach further comprises changing one or more labels for the labelled filtered data based on location speed.

31. The computing device according to any one of claims 27-30, wherein the majority voting approach comprises: comparing the rule-based approach labelled data set with the satellite images approach labelled data set to determine the labels to be used for the filtered data, wherein, for each label: if the label for the filtered data in the rule-based approach labelled data set is the same as the label in the satellite images approach labelled data set, then the label is used; if the label for the filtered data in the rule-based approach labelled data set is different from the label in the satellite images approach labelled data set, then the label that is most frequent from the rule-based approach labelled data set and the satellite images approach labelled data set is used; and if the label for the filtered data in the rule-based approach labelled data set is different from the label in the satellite images approach labelled data set and there is no label that is most frequent, then a weighting is applied and the label with the highest weight is used.

32. The computing device according to any one of claims 18-31, wherein classifying the labeled data, using one of a plurality of machine learning (ML) models to infer the user environment, includes classifying the labelled data as one of: indoor or outdoor for two classes; indoor, outdoor or in-between for three classes; and indoor, outdoor, semi-indoor, or semi-outdoor for four classes.

33. The computing device according to any one of claims 18-32, wherein the one or more communication networks are one or more radio access networks (RANs).

34. The computing device according to any one of claims 18-33, wherein using the classified data to detect user environments in the one or more communication networks includes using the classified data for one or more of:network planning; spectrum strategy; efficient resource utilization; customizing network parameters; customizing network configurations; enhancing user experience; capacity planning; virtual drive testing; performance diagnostics; and performance optimization;35. A computer program (1843) comprising instructions (8244) which, when executed by processing circuitry (1802), causes the processing circuitry to carry out the methods of any one of claims 1-17.

36. A carrier containing a computer program (1843) according to claim 35, wherein the carrier comprises one of an electronic signals, optical signal, radio signal or computer readable storage medium.

37. A computer program product (1841) comprising a non-transitory computer readable medium (1842) having stored thereon a computer program (1843) according to claim 35.

38. An apparatus (1800), the apparatus comprising: a memory (1808); and processing circuitry (1802) coupled to the memory, wherein the apparatus is configured to perform the methods of any one of claims 1-17.

Images